Nov 03 15:59:38.793700 ip-10-0-133-129 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 15:59:38.793711 ip-10-0-133-129 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 15:59:38.793718 ip-10-0-133-129 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 15:59:38.794004 ip-10-0-133-129 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 15:59:49.028147 ip-10-0-133-129 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 15:59:49.028167 ip-10-0-133-129 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot efdbc63018ae43bbb56163a5d3cd46b4 --
Nov 03 16:01:36.211783 ip-10-0-133-129 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 16:01:36.659882 ip-10-0-133-129 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:01:36.659882 ip-10-0-133-129 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 16:01:36.659882 ip-10-0-133-129 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:01:36.659882 ip-10-0-133-129 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:01:36.667296 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667171    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.667296 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667296    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667303    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667309    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667315    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667319    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667323    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667342    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667347    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667351    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667356    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667360    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667364    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667368    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667372    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.667394 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667377    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667402    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667408    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667412    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667417    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667421    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667425    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667429    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667434    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667438    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667441    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667446    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667450    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667454    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667459    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667463    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667467    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667472    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667476    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667480    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.667721 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667484    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667488    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667492    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667496    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667500    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667511    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667518    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667524    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667529    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667534    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667539    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667543    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667546    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667552    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667556    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667561    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667565    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667569    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667573    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.668198 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667578    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667582    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667586    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667591    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667596    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667601    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667607    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667611    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667615    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667619    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667624    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667628    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667633    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667637    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667642    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667646    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667650    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667654    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667659    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.668665 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667664    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667668    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667673    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667676    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667681    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667684    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667689    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667694    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667699    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667703    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667706    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667711    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667715    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667719    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667723    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667727    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667731    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667735    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667740    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.669117 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667744    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667748    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667753    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667756    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667761    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667765    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667769    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667774    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667778    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667782    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667787    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667791    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667795    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.667799    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669027    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669040    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669044    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669049    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669053    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669057    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.669703 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669063    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669067    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669071    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669076    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669081    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669085    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669089    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669093    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669097    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669101    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669105    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669109    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669114    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669118    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669123    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669127    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669131    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669135    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669140    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669144    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.670208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669148    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669152    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669157    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669161    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669165    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669169    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669173    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669184    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669189    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669193    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669198    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669202    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669206    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669210    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669214    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669219    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669224    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669228    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669231    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.670768 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669235    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669240    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669244    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669247    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669251    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669256    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669260    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669264    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669268    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669272    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669276    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669280    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669284    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669288    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669292    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669296    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669301    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669305    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669311    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.671256 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669316    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669320    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669333    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669338    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669342    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669346    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669351    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669356    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669360    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669364    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669368    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669372    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669376    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669381    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669401    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669406    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669409    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669414    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669418    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669422    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.671733 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669426    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669430    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669433    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669437    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669442    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669447    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669451    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669455    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669459    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669464    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669470    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669476    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669480    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669485    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669489    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669496    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669501    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669504    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669509    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.672208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669514    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669518    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.669522    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670297    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670306    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670311    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670315    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670319    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670324    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670328    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670332    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670336    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670340    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670345    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670349    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670353    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670357    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670361    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670365    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670370    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.672659 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670374    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670377    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670397    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670402    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670407    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670412    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670416    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670420    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670428    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670433    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670437    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670442    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670446    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670451    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670455    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670459    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670463    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670467    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670471    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670476    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.673133 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670479    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670485    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670491    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670496    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670503    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670509    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670514    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670520    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670524    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670529    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670533    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670538    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670543    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670548    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670552    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670557    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670560    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670564    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670568    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.673627 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670572    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670576    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670581    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670587    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670591    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670595    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670600    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670604    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670608    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670611    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670615    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670620    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670624    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670628    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670632    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670636    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670641    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670645    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670649    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.674116 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670653    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670657    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670661    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670665    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670669    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670673    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670677    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670681    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670685    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670690    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670694    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670699    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670702    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670707    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670712    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670716    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670725    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670728    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670732    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.674619 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670736    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670741    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670745    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670749    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670753    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670756    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670760    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670765    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670769    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670774    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670778    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.670783    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.671948    2584 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.671964    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.671972    2584 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.671980    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.671988    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.671994    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672001    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672007    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672013    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 16:01:36.675104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672019    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672024    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672029    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672033    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672038    2584 flags.go:64] FLAG: --cgroup-root=""
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672043    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672047    2584 flags.go:64] FLAG: --client-ca-file=""
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672052    2584 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672057    2584 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672066    2584 flags.go:64] FLAG: --cluster-domain=""
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672070    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672076    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672081    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672086    2584 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672092    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672097    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672102    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672107    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672112    2584 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672117    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672121    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672126    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672131    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672138    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 16:01:36.675620 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672143    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672148    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672152    2584 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672157    2584 flags.go:64] FLAG: --enable-server="true"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672161    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672167    2584 flags.go:64] FLAG: --event-burst="100"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672172    2584 flags.go:64] FLAG: --event-qps="50"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672176    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672181    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672185    2584 flags.go:64] FLAG: --eviction-hard=""
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672191    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672196    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672201    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672205    2584 flags.go:64] FLAG: --eviction-soft=""
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672210    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672215    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672219    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672224    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672230    2584 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672235    2584 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672240    2584 flags.go:64] FLAG: --feature-gates=""
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672246    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672252    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672257    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672262    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672266    2584 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 16:01:36.676196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672271    2584 flags.go:64] FLAG: --help="false"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672276    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672281    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672287    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672291    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672296    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672302    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672307    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672312    2584 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672316    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672321    2584 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672325    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672330    2584 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672335    2584 flags.go:64] FLAG: --kube-reserved=""
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672340    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672344    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672349    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672353    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672358    2584 flags.go:64] FLAG: --lock-file=""
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672362    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672366    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672372    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672381    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 16:01:36.676826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672398    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672403    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672409    2584 flags.go:64] FLAG: --logging-format="text"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672414    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672419    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672424    2584 flags.go:64] FLAG: --manifest-url=""
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672429    2584 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672437    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672442    2584 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672448    2584 flags.go:64] FLAG: --max-pods="110"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672453    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672458    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672463    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672468    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672473    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672478    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672482    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672493    2584 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672498    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672503    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672508    2584 flags.go:64] FLAG: --pod-cidr=""
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672512    2584 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672517    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672522    2584 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672526    2584 flags.go:64] FLAG: --port="10250"
Nov 03 16:01:36.677359 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672531    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672535    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-06e3420b04f089dfd"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672541    2584 flags.go:64] FLAG: --qos-reserved=""
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672545    2584 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672550    2584 flags.go:64] FLAG: --register-node="true"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672555    2584 flags.go:64] FLAG: --register-with-taints=""
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672561    2584 flags.go:64] FLAG: --registry-burst="10"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672565    2584 flags.go:64] FLAG: --registry-qps="5"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672570    2584 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672576    2584 flags.go:64] FLAG: --reserved-memory=""
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672584    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672589    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672594    2584 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672599    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672604    2584 flags.go:64] FLAG: --runonce="false"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672609    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672614    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672619    2584 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672623    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672633    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672637    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672642    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672647    2584 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672651    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672656    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 16:01:36.678145 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672660    2584 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672665    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672670    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672675    2584 flags.go:64] FLAG: --system-cgroups=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672680    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672688    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672692    2584 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672697    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672704    2584 flags.go:64] FLAG: --tls-min-version=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672709    2584 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672714    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672718    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672726    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672731    2584 flags.go:64] FLAG: --v="2"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672738    2584 flags.go:64] FLAG: --version="false"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672745    2584 flags.go:64] FLAG: --vmodule=""
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672751    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.672756    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672963    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672969    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672975    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672979    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672984    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.678848 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672988    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672992    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.672997    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673005    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673010    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673014    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673018    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673023    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673027    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673031    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673036    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673039    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673043    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673048    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673053    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673056    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673060    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673066    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673072    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.679441 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673076    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673080    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673086    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673091    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673095    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673100    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673104    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673108    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673113    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673117    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673121    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673125    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673129    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673133    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673138    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673141    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673148    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673152    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673156    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673162    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.679949 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673165    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673169    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673173    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673177    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673182    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673186    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673190    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673194    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673198    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673202    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673206    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673212    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673215    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673220    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673226    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673230    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673234    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673238    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673242    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673246    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.680465 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673250    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673254    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673259    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673264    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673268    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673272    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673276    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673280    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673286    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673290    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673294    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673299    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673303    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673307    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673311    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673315    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673320    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673324    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673328    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.681270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673332    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673336    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673340    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673345    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673349    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673353    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673357    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673362    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673366    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673370    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673374    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673378    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673397    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673403    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673407    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673411    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673416    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673420    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673425    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.682105 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673429    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673434    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673439    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673444    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.673483    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673535    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673542    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673546    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673550    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673556    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673560    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673564    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673569    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673572    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673577    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673581    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.682645 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673585    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673589    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673593    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673598    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673604    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673608    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673612    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673616    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673620    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673624    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673628    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673633    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673637    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673642    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673646    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673651    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673655    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673659    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673666    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.683057 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673674    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673678    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673682    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673686    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673691    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673695    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673699    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673703    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673707    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673712    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673716    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673721    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673725    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673731    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673735    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673740    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673745    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673750    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673754    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.683652 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673758    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673762    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673767    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673771    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673775    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673779    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673784    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673788    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673792    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673797    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673801    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673806    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673810    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673819    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673823    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673828    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673832    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673837    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673841    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673845    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.684107 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673850    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673855    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673859    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673865    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673870    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673874    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673878    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673882    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673886    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673891    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673895    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673899    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673902    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673906    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673910    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673914    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673919    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673923    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673934    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673938    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.684596 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673944    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673948    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673953    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673957    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673961    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673968    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673973    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673978    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673982    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673986    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673990    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673994    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.673998    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.674002    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.674006    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.674010    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.674014    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.683396    2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.683421    2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 16:01:36.685101 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.684163    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.684169    2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684243    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684249    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684253    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684256    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684259    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684262    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684264    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684267    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684271    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684273    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684276    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684278    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684281    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684284    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684286    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684289    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684292    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.685580 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684294    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684299    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684303    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684306    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684309    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684312    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684314    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684317    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684319    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684322    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684325    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684329    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684333    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684336    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684339    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684341    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684344    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684347    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684350    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.686040 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684352    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684355    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684357    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684360    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684363    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684366    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684369    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684371    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684374    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684377    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684401    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684407    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684411    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684415    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684418    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684421    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684423    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684426    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684429    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684432    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.686517 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684435    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684437    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684440    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684443    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684446    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684449    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684451    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684454    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684457    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684459    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684462    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684464    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684467    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684470    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684473    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684476    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684479    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684481    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684484    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.687008 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684487    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684489    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684492    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684495    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684499    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684501    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684504    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684507    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684510    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684512    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684515    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684519    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684521    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684524    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684526    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684529    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684532    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684535    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684538    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684540    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.687615 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684543    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684546    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684549    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684552    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684554    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684557    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684560    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684562    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684565    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684568    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684570    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.684608    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684761    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684766    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684769    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684771    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.688103 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684774    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684777    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684780    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684783    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684785    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684788    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684791    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684794    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684797    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684799    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684802    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684804    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684807    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684810    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684813    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684816    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684818    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684822    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684827    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684830    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.688546 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684833    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684835    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684838    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684841    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684843    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684846    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684848    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684851    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684853    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684856    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684858    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684861    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684863    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684866    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684869    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684872    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684874    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684877    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684881    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.689027 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684884    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684886    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684889    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684892    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684894    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684898    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684900    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684903    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684906    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684908    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684911    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684914    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684917    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684919    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684922    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684924    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684927    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684929    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684932    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684934    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.689488 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684937    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684940    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684942    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684945    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684947    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684950    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684952    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684954    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684958    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684961    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684963    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684966    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684969    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684971    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684974    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684976    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684979    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684982    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684985    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684987    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.689964 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684990    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684992    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684995    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.684997    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685000    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685003    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685006    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685009    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685012    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685014    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685017    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685020    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685022    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685025    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685027    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685030    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685032    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685035    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685037    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.690460 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685039    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685042    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685047    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685051    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.685075    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685110    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685115    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685118    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685121    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685124    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685126    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685130    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685133    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685136    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685138    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:01:36.690911 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685141    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685144    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685148    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685151    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685154    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685157    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685160    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685163    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685166    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685168    2584 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685171    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685174    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685176    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685179    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685182    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685184    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685187    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685189    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685192    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:01:36.691267 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685194    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685197    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685200    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685203    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685205    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685208    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685212    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685215    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685218    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685222    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685225    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685228    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685230    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685233    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685235    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685238    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685241    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685243    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685247    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:01:36.691770 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685249    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685252    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685255    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685257    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685260    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685262    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685265    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685267    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685270    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685273    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685276    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685278    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685281    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685283    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685286    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685289    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685291    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685294    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685297    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:01:36.692208 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685299    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685302    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685304    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685308    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685311    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685313    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685316    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685318    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685321    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685323    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685326    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685329    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685331    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685334    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685337    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685339    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685341    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685344    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685347    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:01:36.692666 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685349    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685351    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685354    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685356    2584 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685359    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685361    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685364    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685366    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685369    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685372    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685375    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685377    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685379    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685382    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685399    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685403    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685406    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685409    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685411    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:36.685414    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:01:36.693121 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.685488    2584 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 16:01:36.761787 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.761759    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 16:01:36.762996 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.762982    2584 server.go:1014] "Starting client certificate rotation"
Nov 03 16:01:36.763108 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.763086    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:01:36.763152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.763143    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:01:36.765396 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.765372    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:01:36.768550 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.768522    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:01:36.778021 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.777988    2584 log.go:25] "Validated CRI v1 runtime API"
Nov 03 16:01:36.782479 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.782460    2584 log.go:25] "Validated CRI v1 image API"
Nov 03 16:01:36.784075 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.784031    2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 16:01:36.787098 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.787070    2584 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8170ca1b-4cfa-4313-bc63-5435016a1192:/dev/nvme0n1p4 a3b99ce7-fd0c-47d3-ba86-fbd6b4a8eedc:/dev/nvme0n1p3]
Nov 03 16:01:36.787197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.787093    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 16:01:36.792591 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.792450    2584 manager.go:217] Machine: {Timestamp:2026-11-03 16:01:36.790850566 +0000 UTC m=+0.462153165 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100436 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22dde31f7a1df68c4fd6dd8956b349 SystemUUID:ec22dde3-1f7a-1df6-8c4f-d6dd8956b349 BootID:efdbc630-18ae-43bb-b561-63a5d3cd46b4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f0:5f:ee:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f0:5f:ee:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:15:f4:50:f4:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 16:01:36.792591 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.792575    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 16:01:36.792772 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.792753    2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 16:01:36.797556 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.797286    2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 16:01:36.797705 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.797555    2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-129.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 16:01:36.798444 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.798433    2584 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 16:01:36.798483 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.798447    2584 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 16:01:36.798483 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.798477    2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:01:36.798540 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.798494    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:01:36.799845 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.799834    2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 16:01:36.802103 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.802083    2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 16:01:36.804666 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804651    2584 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 16:01:36.804805 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804795    2584 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 16:01:36.804848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804830    2584 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 16:01:36.804848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804841    2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 16:01:36.804908 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804852    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 16:01:36.804908 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804862    2584 kubelet.go:405] "Adding apiserver pod source"
Nov 03 16:01:36.804908 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.804871    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 16:01:36.805266 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.805249    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:01:36.807933 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.807911    2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 16:01:36.814799 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.814778    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:01:36.815810 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.815797    2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 16:01:36.815863 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.815828    2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 16:01:36.815998 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.815988    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 16:01:36.816038 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816004    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 16:01:36.816038 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816010    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 16:01:36.816038 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816016    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 16:01:36.816038 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816022    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 16:01:36.816038 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816029    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 16:01:36.816692 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816683    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 16:01:36.816692 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816692    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 16:01:36.816746 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816701    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 16:01:36.816746 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816708    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 16:01:36.816746 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816717    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 16:01:36.816746 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.816726    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 16:01:36.817450 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.817440    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 16:01:36.817450 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.817450    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 16:01:36.821619 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.821602    2584 server.go:1263] "Started kubelet"
Nov 03 16:01:36.821729 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.821682    2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 16:01:36.821782 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.821722    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 16:01:36.821845 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.821816    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 16:01:36.822436 ip-10-0-133-129 systemd[1]: Started Kubernetes Kubelet.
Nov 03 16:01:36.822933 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.822867    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-129.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 16:01:36.823876 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.823858    2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 16:01:36.823949 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.823907    2584 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 16:01:36.828282 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.828258    2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 16:01:36.831680 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.831658    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 16:01:36.832332 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.832320    2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 16:01:36.832466 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.832434    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:36.833162 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833143    2584 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 16:01:36.833162 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833162    2584 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 16:01:36.833289 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833267    2584 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 16:01:36.833289 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833275    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 16:01:36.833289 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833280    2584 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 16:01:36.833488 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833415    2584 factory.go:55] Registering systemd factory
Nov 03 16:01:36.833488 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833431    2584 factory.go:223] Registration of the systemd container factory successfully
Nov 03 16:01:36.833859 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833742    2584 factory.go:153] Registering CRI-O factory
Nov 03 16:01:36.833859 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833757    2584 factory.go:223] Registration of the crio container factory successfully
Nov 03 16:01:36.833859 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833808    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 16:01:36.833859 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833825    2584 factory.go:103] Registering Raw factory
Nov 03 16:01:36.833859 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.833839    2584 manager.go:1199] Started watching for new ooms in manager
Nov 03 16:01:36.835047 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.835031    2584 manager.go:319] Starting recovery of all containers
Nov 03 16:01:36.836186 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.836152    2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-129.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 16:01:36.836379 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.836339    2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-129.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 16:01:36.836817 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.836792    2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 16:01:36.840518 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.836846    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-129.us-east-2.compute.internal.18e4946134f5e938  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-129.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-129.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 16:01:36.821569848 +0000 UTC m=+0.492872446,LastTimestamp:2026-11-03 16:01:36.821569848 +0000 UTC m=+0.492872446,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-129.us-east-2.compute.internal,}"
Nov 03 16:01:36.844859 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.844831    2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 16:01:36.851487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.851462    2584 manager.go:324] Recovery completed
Nov 03 16:01:36.855643 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.855627    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:36.856481 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.856463    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zbpm"
Nov 03 16:01:36.858020 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858000    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:36.858090 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858035    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:36.858090 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858052    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:36.858605 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858590    2584 cpu_manager.go:226] "Starting" policy="none"
Nov 03 16:01:36.858605 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858602    2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 16:01:36.858706 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858620    2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 16:01:36.858706 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.858639    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:36.859428 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.859415    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:36.859486 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.859440    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:36.859486 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.859450    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:36.860800 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.860688    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-129.us-east-2.compute.internal.18e49461372216d1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-129.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-129.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-129.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 16:01:36.858019537 +0000 UTC m=+0.529322137,LastTimestamp:2026-11-03 16:01:36.858019537 +0000 UTC m=+0.529322137,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-129.us-east-2.compute.internal,}"
Nov 03 16:01:36.860966 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.860955    2584 policy_none.go:50] "Start"
Nov 03 16:01:36.860998 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.860970    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 16:01:36.860998 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.860980    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 16:01:36.863051 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.863040    2584 policy_none.go:44] "Start"
Nov 03 16:01:36.866733 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.866705    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zbpm"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.904515    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.905734    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.905756    2584 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.905776    2584 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.905807    2584 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.908140    2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.910203    2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.910911    2584 manager.go:341] "Starting Device Plugin manager"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.911511    2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.911525    2584 server.go:87] "Starting device plugin registration server"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.911769    2584 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.911779    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.911885    2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.911969    2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:36.911977    2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.912568    2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 16:01:36.916140 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:36.912614    2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.006757 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.006563    2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"]
Nov 03 16:01:37.006901 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.006830    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:37.008444 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.008424    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:37.008560 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.008459    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:37.008560 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.008471    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:37.010934 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.010916    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:37.011086 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.011070    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.011123 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.011115    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:37.011920 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.011904    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:37.012270 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012254    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:37.012333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012286    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:37.012333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012297    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:37.012333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012257    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:37.012477 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012344    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:37.012477 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012360    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:37.012595 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012574    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:37.012709 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012607    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:37.012709 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012625    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:37.012709 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.012651    2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.014707 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.014681    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.014827 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.014717    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:01:37.015496 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.015466    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:01:37.015586 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.015509    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:01:37.015586 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.015521    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:01:37.020888 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.020871    2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.020970 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.020897    2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-129.us-east-2.compute.internal\": node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.047566 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.047540    2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.050260 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.050237    2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.103100 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.103061    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.134062 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.134028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19bdd4315736b0c6d37ff28999a14800-config\") pod \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"19bdd4315736b0c6d37ff28999a14800\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.134062 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.134064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.134226 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.134084    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.203376 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.203348    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.234950 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.234922    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.234950 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.234953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.235144 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.234975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19bdd4315736b0c6d37ff28999a14800-config\") pod \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"19bdd4315736b0c6d37ff28999a14800\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.235144 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.235005    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19bdd4315736b0c6d37ff28999a14800-config\") pod \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"19bdd4315736b0c6d37ff28999a14800\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.235144 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.235006    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.235144 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.235019    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.306197 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.306150    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.353148 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.353105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.355476 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.355442    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:37.406556 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.406503    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.509802 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.509710    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.615200 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.615167    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.721184 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.721143    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.762317 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.762248    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 16:01:37.762481 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.762460    2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 16:01:37.821828 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.821792    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:37.872646 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.872590    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:56:37 +0000 UTC" deadline="2028-05-31 03:39:34.5869844 +0000 UTC"
Nov 03 16:01:37.872646 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:37.872642    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13787h37m56.714345738s"
Nov 03 16:01:37.922305 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:37.922264    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:38.022854 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.022771    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:38.036639 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.036601    2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:38.123875 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.123820    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:01:38.168347 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:38.168303    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0d794832c64c25bfb1086aaf2b1f2b.slice/crio-e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69 WatchSource:0}: Error finding container e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69: Status 404 returned error can't find the container with id e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69
Nov 03 16:01:38.168594 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:38.168572    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bdd4315736b0c6d37ff28999a14800.slice/crio-97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f WatchSource:0}: Error finding container 97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f: Status 404 returned error can't find the container with id 97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f
Nov 03 16:01:38.173926 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.173905    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:01:38.214577 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.214546    2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:38.233432 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.233380    2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:38.249693 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.249667    2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:01:38.251887 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.251870    2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:01:38.262573 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.262548    2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:01:38.326128 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.326061    2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:38.808196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.808103    2584 apiserver.go:52] "Watching apiserver"
Nov 03 16:01:38.817641 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.817612    2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 16:01:38.818074 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.818052    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-wblql","openshift-ovn-kubernetes/ovnkube-node-h45wd","kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-ww48r","openshift-dns/node-resolver-mlccd","openshift-image-registry/node-ca-5wvsc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-rtlt9","kube-system/konnectivity-agent-r59q4","openshift-multus/multus-hr4sq","openshift-multus/network-metrics-daemon-njvgc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4","openshift-multus/multus-additional-cni-plugins-zpddq","openshift-network-diagnostics/network-check-target-kdkkw"]
Nov 03 16:01:38.821148 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.820717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.823329 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.823305    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.825798 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.825766    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.825910 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.825861    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 16:01:38.825977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.825957    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.826172 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.826148    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qk9pq\""
Nov 03 16:01:38.826302 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.826251    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 16:01:38.826365 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.826354    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:38.826466 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.826434    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:38.827576 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.827552    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 16:01:38.827676 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.827660    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dchj7\""
Nov 03 16:01:38.827743 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.827699    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 16:01:38.829051 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.828629    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.831117 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.831091    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.832119 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.832062    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 16:01:38.833255 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.833234    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.835985 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.835960    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.838546 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.838523    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:38.839644 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.839609    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:38.842613 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysconfig\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.842735 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842652    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.842735 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842696    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-lib-modules\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.842845 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3bb7151-48a1-4b4d-9784-31da848ac4f0-konnectivity-ca\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.842845 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-socket-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.842955 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-multus-certs\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.842955 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-system-cni-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.843059 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.842966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:38.843235 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-run\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.843348 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30fac5-ac40-4179-9628-fa3203ddb77b-tmp-dir\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.843348 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4vs\" (UniqueName: \"kubernetes.io/projected/6a30fac5-ac40-4179-9628-fa3203ddb77b-kube-api-access-vw4vs\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.843348 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843333    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.843556 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-device-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.843556 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.843659 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843561    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.843659 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.843659 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-multus-daemon-config\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.843820 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843688    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.843820 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.843935 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-k8s-cni-cncf-io\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.843935 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-netns\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.843935 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-kubelet\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.844081 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.843962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-etc-kubernetes\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.844081 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-systemd\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.844081 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-tmp\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.844286 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844146    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3bb7151-48a1-4b4d-9784-31da848ac4f0-agent-certs\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.844925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845020 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844942    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-sys\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845020 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-var-lib-kubelet\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845020 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.844993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a30fac5-ac40-4179-9628-fa3203ddb77b-hosts-file\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.845020 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-sys-fs\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-socket-dir-parent\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845071    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djljv\" (UniqueName: \"kubernetes.io/projected/b903185c-6b3b-45dd-aac2-caaa87d0b075-kube-api-access-djljv\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-registration-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kube-api-access-gmrnn\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74rn\" (UniqueName: \"kubernetes.io/projected/fe4abe85-5418-426a-bb01-57b63c1f5c52-kube-api-access-g74rn\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.845204 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-cnibin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845224    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-bin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-hostroot\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845305    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-cnibin\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-host\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845354    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-system-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845378    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-cni-binary-copy\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-conf-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-modprobe-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845447    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-kubernetes\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.845548 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845529    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-conf\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.846071 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-tuned\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.846071 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845592    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-multus\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.846071 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845642    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9v5s\" (UniqueName: \"kubernetes.io/projected/a467d9db-7d44-47e3-a142-30cb089a882b-kube-api-access-k9v5s\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.846071 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.845784    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:38.849624 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.849604    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:38.853998 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.853976    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:38.854110 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.854039    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:38.856805 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.856773    2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 16:01:38.912769 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.912708    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"19bdd4315736b0c6d37ff28999a14800","Type":"ContainerStarted","Data":"97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f"}
Nov 03 16:01:38.915377 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.915339    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerStarted","Data":"e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69"}
Nov 03 16:01:38.925598 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.925563    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wxz4"
Nov 03 16:01:38.934671 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.934635    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 16:01:38.935046 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935013    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.935120 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935070    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 16:01:38.935212 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935196    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.935520 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935499    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 16:01:38.935614 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935559    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.935614 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935587    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 16:01:38.935614 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935593    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.935772 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935637    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.935772 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.935646    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.936207 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936189    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 16:01:38.936385 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936366    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.936486 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936465    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.936563 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936522    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drdlv\""
Nov 03 16:01:38.936618 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936583    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 16:01:38.936711 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936695    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.936774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936743    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.936840 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936824    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 16:01:38.936897 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.936883    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 16:01:38.937130 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.937114    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 16:01:38.937778 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.937761    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c56qh\""
Nov 03 16:01:38.937875 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.937850    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2h492\""
Nov 03 16:01:38.937952 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.937909    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 16:01:38.938008 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.937966    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 16:01:38.938065 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.938017    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5rhh2\""
Nov 03 16:01:38.938849 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.938813    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgwbg\""
Nov 03 16:01:38.939331 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.939297    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 16:01:38.939423 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.939366    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r589\""
Nov 03 16:01:38.939423 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.939368    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jdvsk\""
Nov 03 16:01:38.940265 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.940240    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wxz4"
Nov 03 16:01:38.946528 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a30fac5-ac40-4179-9628-fa3203ddb77b-hosts-file\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.946657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946538    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.946657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-sys\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.946657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-var-lib-kubelet\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.946657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946594    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a30fac5-ac40-4179-9628-fa3203ddb77b-hosts-file\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.946657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-dbus\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:38.946657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-systemd-units\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946662    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-sys\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946672    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovn-node-metrics-cert\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklmt\" (UniqueName: \"kubernetes.io/projected/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-kube-api-access-dklmt\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-socket-dir-parent\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9ch\" (UniqueName: \"kubernetes.io/projected/1c0598b6-8a65-4e35-9347-907aa42e8781-kube-api-access-2l9ch\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-var-lib-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-script-lib\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30fac5-ac40-4179-9628-fa3203ddb77b-tmp-dir\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-cnibin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-host\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.946977 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946965    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-host-slash\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946990    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl27k\" (UniqueName: \"kubernetes.io/projected/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-kube-api-access-kl27k\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947017    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-cni-binary-copy\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-conf-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-modprobe-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.946794    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-kubernetes\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-tuned\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-netns\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.947645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947539    2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 16:01:38.948079 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-ovn\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.948079 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947786    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-cni-binary-copy\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948079 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947875    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-multus\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948079 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947929    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-conf-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948079 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948042    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-modprobe-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948092    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-kubernetes\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948102    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30fac5-ac40-4179-9628-fa3203ddb77b-tmp-dir\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948132    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-var-lib-kubelet\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.947791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-multus\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948179    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948255    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-cnibin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948289    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-host\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.948333 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-lib-modules\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948357    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-socket-dir-parent\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-bin\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948500    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-lib-modules\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-multus-certs\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948559    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-multus-certs\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948595    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:38.948748 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-iptables-alerter-script\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-node-log\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-env-overrides\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948872    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-k8s-cni-cncf-io\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948879    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948899    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-kubelet\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948950    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-kubelet\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.948989    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-k8s-cni-cncf-io\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-etc-kubernetes\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-tmp\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.949138 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949128    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-etc-kubernetes\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3bb7151-48a1-4b4d-9784-31da848ac4f0-agent-certs\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949190    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66a8e707-3e0b-42f2-988f-517d834a869c-serviceca\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-kubelet\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949222    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-netd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-slash\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djljv\" (UniqueName: \"kubernetes.io/projected/b903185c-6b3b-45dd-aac2-caaa87d0b075-kube-api-access-djljv\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949299    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-registration-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.949658 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949373    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kube-api-access-gmrnn\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.949926 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949734    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.949926 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949801    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-registration-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.950033 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949938    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.950033 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.949985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-sys-fs\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.950033 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74rn\" (UniqueName: \"kubernetes.io/projected/fe4abe85-5418-426a-bb01-57b63c1f5c52-kube-api-access-g74rn\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.950190 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950082    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-sys-fs\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.950190 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-bin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950190 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-hostroot\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950190 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-cnibin\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.950384 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxvl\" (UniqueName: \"kubernetes.io/projected/66a8e707-3e0b-42f2-988f-517d834a869c-kube-api-access-drxvl\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:38.950384 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950235    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.950384 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-system-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950384 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950299    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-hostroot\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950384 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-conf\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.950384 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-config\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9v5s\" (UniqueName: \"kubernetes.io/projected/a467d9db-7d44-47e3-a142-30cb089a882b-kube-api-access-k9v5s\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950424    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-system-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950481    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-bin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysconfig\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3bb7151-48a1-4b4d-9784-31da848ac4f0-konnectivity-ca\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950556    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-conf\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950663    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-socket-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.950700 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-socket-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-kubelet-config\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-systemd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950766    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-cnibin\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950790    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-system-cni-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-run\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950823    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysconfig\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4vs\" (UniqueName: \"kubernetes.io/projected/6a30fac5-ac40-4179-9628-fa3203ddb77b-kube-api-access-vw4vs\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950868    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-system-cni-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-device-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66a8e707-3e0b-42f2-988f-517d834a869c-host\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.950979    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-etc-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951008    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-multus-daemon-config\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.951197 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951129    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-log-socket\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:38.952013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.952013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-netns\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.952013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951264    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-systemd\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.952013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951365    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-systemd\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.952013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951416    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-run\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.952013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.951426    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3bb7151-48a1-4b4d-9784-31da848ac4f0-konnectivity-ca\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.952322 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952019    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.952322 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952054    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-multus-daemon-config\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.952322 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952111    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-device-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.952322 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952205    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-netns\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.952553 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952414    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.952553 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952538    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.952819 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.952797    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.953857 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.953831    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3bb7151-48a1-4b4d-9784-31da848ac4f0-agent-certs\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:38.955457 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.955423    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-tuned\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.956064 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.956043    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-tmp\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.960445 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.960417    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:38.960590 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.960448    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:38.961358 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.961334    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8262d for pod openshift-network-diagnostics/network-check-target-kdkkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:38.961487 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:38.961456    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d podName:e3c566eb-b54b-47e7-a848-c1cacacc63e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:39.46142894 +0000 UTC m=+3.132731543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8262d" (UniqueName: "kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d") pod "network-check-target-kdkkw" (UID: "e3c566eb-b54b-47e7-a848-c1cacacc63e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:38.966899 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.966867    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djljv\" (UniqueName: \"kubernetes.io/projected/b903185c-6b3b-45dd-aac2-caaa87d0b075-kube-api-access-djljv\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:38.967057 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.966904    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kube-api-access-gmrnn\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:38.970285 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.970248    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74rn\" (UniqueName: \"kubernetes.io/projected/fe4abe85-5418-426a-bb01-57b63c1f5c52-kube-api-access-g74rn\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:38.970459 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.970436    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9v5s\" (UniqueName: \"kubernetes.io/projected/a467d9db-7d44-47e3-a142-30cb089a882b-kube-api-access-k9v5s\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:38.973205 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:38.973174    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4vs\" (UniqueName: \"kubernetes.io/projected/6a30fac5-ac40-4179-9628-fa3203ddb77b-kube-api-access-vw4vs\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:39.051947 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.051908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-netns\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.051947 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.051954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-ovn\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.051981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-bin\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-iptables-alerter-script\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-node-log\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-env-overrides\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66a8e707-3e0b-42f2-988f-517d834a869c-serviceca\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-kubelet\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-netd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052215 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-slash\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxvl\" (UniqueName: \"kubernetes.io/projected/66a8e707-3e0b-42f2-988f-517d834a869c-kube-api-access-drxvl\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-config\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-kubelet-config\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-systemd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052390    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66a8e707-3e0b-42f2-988f-517d834a869c-host\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-etc-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052431    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-log-socket\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-dbus\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-systemd-units\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovn-node-metrics-cert\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklmt\" (UniqueName: \"kubernetes.io/projected/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-kube-api-access-dklmt\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9ch\" (UniqueName: \"kubernetes.io/projected/1c0598b6-8a65-4e35-9347-907aa42e8781-kube-api-access-2l9ch\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:39.052674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-var-lib-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-script-lib\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-host-slash\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.052685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl27k\" (UniqueName: \"kubernetes.io/projected/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-kube-api-access-kl27k\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.052827    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.052900    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret podName:d3387caf-55fb-4a86-9b89-f38c8fbe1915 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:39.552884121 +0000 UTC m=+3.224186751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret") pod "global-pull-secret-syncer-rtlt9" (UID: "d3387caf-55fb-4a86-9b89-f38c8fbe1915") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053019    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-netns\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053064    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-ovn\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.053124    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:39.053168 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.053153    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:39.553144481 +0000 UTC m=+3.224447067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:39.053438 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053177    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-bin\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053547    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-systemd-units\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053563    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-netd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053601    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-config\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053620    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-node-log\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053645    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-systemd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053669    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-var-lib-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053659    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-log-socket\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053690    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-iptables-alerter-script\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053743    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-dbus\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053757    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053770    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053808    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-kubelet\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053880    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-kubelet-config\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053907    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-slash\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053942    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-host-slash\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053966    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.053987    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-etc-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.054122 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.054013    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66a8e707-3e0b-42f2-988f-517d834a869c-host\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.055124 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.054074    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-env-overrides\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.055124 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.054230    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66a8e707-3e0b-42f2-988f-517d834a869c-serviceca\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.056499 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.056458    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-script-lib\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.057053 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.057029    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovn-node-metrics-cert\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.072848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.072767    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9ch\" (UniqueName: \"kubernetes.io/projected/1c0598b6-8a65-4e35-9347-907aa42e8781-kube-api-access-2l9ch\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:39.072848 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.072823    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl27k\" (UniqueName: \"kubernetes.io/projected/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-kube-api-access-kl27k\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.073601 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.073576    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklmt\" (UniqueName: \"kubernetes.io/projected/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-kube-api-access-dklmt\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.076057 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.076032    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxvl\" (UniqueName: \"kubernetes.io/projected/66a8e707-3e0b-42f2-988f-517d834a869c-kube-api-access-drxvl\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.101881 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.101852    2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:01:39.137321 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.137281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hr4sq"
Nov 03 16:01:39.151635 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.151593    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda467d9db_7d44_47e3_a142_30cb089a882b.slice/crio-557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625 WatchSource:0}: Error finding container 557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625: Status 404 returned error can't find the container with id 557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625
Nov 03 16:01:39.163617 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.163579    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:01:39.168196 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.168173    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:01:39.168324 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.168287    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:01:39.176897 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.176862    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb903185c_6b3b_45dd_aac2_caaa87d0b075.slice/crio-18cbf6bd749c66f42041c1d2caf3ee04ffdfa163d5a0e56ea97f2120a23a75a4 WatchSource:0}: Error finding container 18cbf6bd749c66f42041c1d2caf3ee04ffdfa163d5a0e56ea97f2120a23a75a4: Status 404 returned error can't find the container with id 18cbf6bd749c66f42041c1d2caf3ee04ffdfa163d5a0e56ea97f2120a23a75a4
Nov 03 16:01:39.182112 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.182082    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a30fac5_ac40_4179_9628_fa3203ddb77b.slice/crio-3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c WatchSource:0}: Error finding container 3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c: Status 404 returned error can't find the container with id 3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c
Nov 03 16:01:39.184831 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.184802    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:01:39.185047 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.185016    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4abe85_5418_426a_bb01_57b63c1f5c52.slice/crio-751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6 WatchSource:0}: Error finding container 751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6: Status 404 returned error can't find the container with id 751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6
Nov 03 16:01:39.187633 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.187608    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:39.203457 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.203204    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78aeb2f_1c43_48f7_8b3e_4f0f9a795198.slice/crio-4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973 WatchSource:0}: Error finding container 4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973: Status 404 returned error can't find the container with id 4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973
Nov 03 16:01:39.213432 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.213382    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:01:39.217643 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.217433    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:01:39.218916 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.218894    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:39.230874 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.230834    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ca2e7f_7ccb_4b0e_a735_21e7d20791d7.slice/crio-1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4 WatchSource:0}: Error finding container 1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4: Status 404 returned error can't find the container with id 1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4
Nov 03 16:01:39.233446 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.233417    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a8e707_3e0b_42f2_988f_517d834a869c.slice/crio-085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8 WatchSource:0}: Error finding container 085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8: Status 404 returned error can't find the container with id 085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8
Nov 03 16:01:39.237789 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:01:39.237727    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5dea52a_10d2_4ca8_a4e9_096f66ad73fc.slice/crio-ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860 WatchSource:0}: Error finding container ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860: Status 404 returned error can't find the container with id ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860
Nov 03 16:01:39.558703 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.558612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:39.558703 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.558673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.558728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558797    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558816    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558824    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558899    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:40.558881139 +0000 UTC m=+4.230183726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558899    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:39.558948 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558952    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret podName:d3387caf-55fb-4a86-9b89-f38c8fbe1915 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:40.558934435 +0000 UTC m=+4.230237040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret") pod "global-pull-secret-syncer-rtlt9" (UID: "d3387caf-55fb-4a86-9b89-f38c8fbe1915") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:39.559229 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558828    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8262d for pod openshift-network-diagnostics/network-check-target-kdkkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:39.559229 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:39.558987    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d podName:e3c566eb-b54b-47e7-a848-c1cacacc63e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:40.558980219 +0000 UTC m=+4.230282845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8262d" (UniqueName: "kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d") pod "network-check-target-kdkkw" (UID: "e3c566eb-b54b-47e7-a848-c1cacacc63e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:39.923364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.923204    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r59q4" event={"ID":"f3bb7151-48a1-4b4d-9784-31da848ac4f0","Type":"ContainerStarted","Data":"d9f3a0675540a229942cbffce98a4240752b7560cf2fba953a01a8095eaf655a"}
Nov 03 16:01:39.926316 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.926205    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4" event={"ID":"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198","Type":"ContainerStarted","Data":"4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973"}
Nov 03 16:01:39.928319 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.928286    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mlccd" event={"ID":"6a30fac5-ac40-4179-9628-fa3203ddb77b","Type":"ContainerStarted","Data":"3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c"}
Nov 03 16:01:39.931072 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.930966    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860"}
Nov 03 16:01:39.933719 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.933686    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5wvsc" event={"ID":"66a8e707-3e0b-42f2-988f-517d834a869c","Type":"ContainerStarted","Data":"085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8"}
Nov 03 16:01:39.943755 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.943702    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:56:39 +0000 UTC" deadline="2028-05-22 04:45:26.378552208 +0000 UTC"
Nov 03 16:01:39.943755 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.943752    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13572h43m46.434804975s"
Nov 03 16:01:39.949719 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.949610    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wblql" event={"ID":"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7","Type":"ContainerStarted","Data":"1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4"}
Nov 03 16:01:39.958103 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.957072    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerStarted","Data":"751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6"}
Nov 03 16:01:39.959947 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.959850    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ww48r" event={"ID":"b903185c-6b3b-45dd-aac2-caaa87d0b075","Type":"ContainerStarted","Data":"18cbf6bd749c66f42041c1d2caf3ee04ffdfa163d5a0e56ea97f2120a23a75a4"}
Nov 03 16:01:39.961595 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:39.961563    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hr4sq" event={"ID":"a467d9db-7d44-47e3-a142-30cb089a882b","Type":"ContainerStarted","Data":"557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625"}
Nov 03 16:01:40.568246 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.568200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:40.568421 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.568256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:40.568421 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.568308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:40.568573 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.568463    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:40.568573 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.568531    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret podName:d3387caf-55fb-4a86-9b89-f38c8fbe1915 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:42.568511604 +0000 UTC m=+6.239814198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret") pod "global-pull-secret-syncer-rtlt9" (UID: "d3387caf-55fb-4a86-9b89-f38c8fbe1915") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:40.569099 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.569075    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:40.569203 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.569139    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:42.569121886 +0000 UTC m=+6.240424499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:40.569283 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.569229    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:40.569283 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.569247    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:40.569283 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.569260    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8262d for pod openshift-network-diagnostics/network-check-target-kdkkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:40.569437 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.569292    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d podName:e3c566eb-b54b-47e7-a848-c1cacacc63e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:42.569282397 +0000 UTC m=+6.240585001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8262d" (UniqueName: "kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d") pod "network-check-target-kdkkw" (UID: "e3c566eb-b54b-47e7-a848-c1cacacc63e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:40.907143 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.906859    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:40.907322 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.907294    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:40.907776 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.907753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:40.907899 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.907847    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:40.907963 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.907919    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:40.908029 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:40.908010    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:40.944704 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.944656    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:56:39 +0000 UTC" deadline="2028-04-24 04:24:39.686784165 +0000 UTC"
Nov 03 16:01:40.944704 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.944703    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12900h22m58.742088401s"
Nov 03 16:01:40.978970 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.978929    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"19bdd4315736b0c6d37ff28999a14800","Type":"ContainerStarted","Data":"7010e745dc679c9b46a220cc59e1e8b31c52be781d588332f7a9e2e2dd6601c0"}
Nov 03 16:01:40.989824 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:40.989785    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerStarted","Data":"fcf1749bd1b9b6d4d0181faa067aa7bffb36e026b29ec797f2202cc6af742cbc"}
Nov 03 16:01:41.078759 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:41.078600    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal" podStartSLOduration=3.078576178 podStartE2EDuration="3.078576178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:41.056404459 +0000 UTC m=+4.727707085" watchObservedRunningTime="2026-11-03 16:01:41.078576178 +0000 UTC m=+4.749878794"
Nov 03 16:01:41.996695 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:41.996658    2584 generic.go:356] "Generic (PLEG): container finished" podID="2e0d794832c64c25bfb1086aaf2b1f2b" containerID="fcf1749bd1b9b6d4d0181faa067aa7bffb36e026b29ec797f2202cc6af742cbc" exitCode=0
Nov 03 16:01:41.997678 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:41.997652    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerDied","Data":"fcf1749bd1b9b6d4d0181faa067aa7bffb36e026b29ec797f2202cc6af742cbc"}
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:42.589297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:42.589351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:42.589387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.589540    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.589605    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret podName:d3387caf-55fb-4a86-9b89-f38c8fbe1915 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:46.589586135 +0000 UTC m=+10.260888747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret") pod "global-pull-secret-syncer-rtlt9" (UID: "d3387caf-55fb-4a86-9b89-f38c8fbe1915") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.590024    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.590076    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:46.590060255 +0000 UTC m=+10.261362846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.590158    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.590174    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.590188    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8262d for pod openshift-network-diagnostics/network-check-target-kdkkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:42.590270 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.590229    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d podName:e3c566eb-b54b-47e7-a848-c1cacacc63e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:46.590215941 +0000 UTC m=+10.261518539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8262d" (UniqueName: "kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d") pod "network-check-target-kdkkw" (UID: "e3c566eb-b54b-47e7-a848-c1cacacc63e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:42.908662 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:42.908573    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:42.908843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.908738    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:42.909191 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:42.909167    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:42.909296 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.909271    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:42.909369 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:42.909353    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:42.909490 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:42.909466    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:43.020841 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:43.020071    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerStarted","Data":"002978c047f969738204c2315c4efe492f640f899f78941f6ea3e1daf0332f05"}
Nov 03 16:01:43.027063 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:43.026315    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="e8736a45f87f48e4fc5f162c19fd5195971f204c9917e88c9dd2901ebd3eb8e2" exitCode=0
Nov 03 16:01:43.027063 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:43.026361    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerDied","Data":"e8736a45f87f48e4fc5f162c19fd5195971f204c9917e88c9dd2901ebd3eb8e2"}
Nov 03 16:01:43.083662 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:43.082295    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" podStartSLOduration=5.082274771 podStartE2EDuration="5.082274771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:43.051259041 +0000 UTC m=+6.722561654" watchObservedRunningTime="2026-11-03 16:01:43.082274771 +0000 UTC m=+6.753577385"
Nov 03 16:01:44.043675 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:44.043635    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r59q4" event={"ID":"f3bb7151-48a1-4b4d-9784-31da848ac4f0","Type":"ContainerStarted","Data":"ac8532d24cf6306e102d3d3163b500b952f4cf97f10027755086630d9fe34d12"}
Nov 03 16:01:44.909808 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:44.909242    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:44.909808 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:44.909401    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:44.909808 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:44.909553    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:44.909808 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:44.909643    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:44.909808 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:44.909696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:44.909808 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:44.909768    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:45.048650 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:45.048550    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5wvsc" event={"ID":"66a8e707-3e0b-42f2-988f-517d834a869c","Type":"ContainerStarted","Data":"865517f3cc69ae54d4e4fae488badb0720ad66ea070d26d88a9c5b944f2f0bfe"}
Nov 03 16:01:45.072319 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:45.072260    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r59q4" podStartSLOduration=3.730159413 podStartE2EDuration="8.072242359s" totalImagesPullingTime="4.342082946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:44.077976372 +0000 UTC m=+7.749278984" watchObservedRunningTime="2026-11-03 16:01:45.072242359 +0000 UTC m=+8.743545041"
Nov 03 16:01:46.055447 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.055344    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4" event={"ID":"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198","Type":"ContainerStarted","Data":"0f8be689d448bcd976eac56567eb5a7a33364b0a7cab0386c49459d103e5c7cd"}
Nov 03 16:01:46.630584 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.630543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:46.630763 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.630618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:46.630763 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.630647    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:46.630893 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.630810    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:46.630893 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.630828    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:46.630893 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.630840    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8262d for pod openshift-network-diagnostics/network-check-target-kdkkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:46.631040 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.630894    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d podName:e3c566eb-b54b-47e7-a848-c1cacacc63e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:54.630879994 +0000 UTC m=+18.302182599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8262d" (UniqueName: "kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d") pod "network-check-target-kdkkw" (UID: "e3c566eb-b54b-47e7-a848-c1cacacc63e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:46.631310 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.631285    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:46.631412 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.631353    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret podName:d3387caf-55fb-4a86-9b89-f38c8fbe1915 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:54.631336329 +0000 UTC m=+18.302638919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret") pod "global-pull-secret-syncer-rtlt9" (UID: "d3387caf-55fb-4a86-9b89-f38c8fbe1915") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:46.631470 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.631442    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:46.631539 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.631478    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:54.631466673 +0000 UTC m=+18.302769264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:46.908787 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.906926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:46.908787 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.907076    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:46.908787 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.907517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:46.908787 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.907624    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:46.908787 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:46.907697    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:46.908787 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:46.907779    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:47.346514 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.346439    2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:47.347568 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.347546    2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:47.367230 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.366557    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5wvsc" podStartSLOduration=3.908603381 podStartE2EDuration="9.366535609s" totalImagesPullingTime="5.457932228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:45.07261189 +0000 UTC m=+8.743914502" watchObservedRunningTime="2026-11-03 16:01:47.366535609 +0000 UTC m=+11.037838218"
Nov 03 16:01:47.545106 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.545079    2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 16:01:47.917774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.917650    2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:01:47.545097521Z","UUID":"5e728b64-1242-4772-8a13-213c8a717422","Handler":null,"Name":"","Endpoint":""}
Nov 03 16:01:47.919978 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.919532    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 16:01:47.919978 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:47.919567    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 16:01:48.064890 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.064038    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="e17507f450838d60c3e71816609193023e4ed8b4b3104c473041a0b05837772e" exitCode=0
Nov 03 16:01:48.064890 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.064233    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerDied","Data":"e17507f450838d60c3e71816609193023e4ed8b4b3104c473041a0b05837772e"}
Nov 03 16:01:48.067132 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.067058    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hr4sq" event={"ID":"a467d9db-7d44-47e3-a142-30cb089a882b","Type":"ContainerStarted","Data":"a783373ed5f2d8e6ea53e05d23a71666b0576d02572f126d2f456afe8d61e9e6"}
Nov 03 16:01:48.069939 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.069904    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4" event={"ID":"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198","Type":"ContainerStarted","Data":"7d4d6080158cc24a006ccb50b1c1553503113f0080152b0782ba426661167f35"}
Nov 03 16:01:48.071796 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.071769    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mlccd" event={"ID":"6a30fac5-ac40-4179-9628-fa3203ddb77b","Type":"ContainerStarted","Data":"b398586c9780ff17bed0d9cc28dd46d96165bcbc74f317a752b9920b3abf925f"}
Nov 03 16:01:48.202309 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.201788    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mlccd" podStartSLOduration=2.482460252 podStartE2EDuration="11.201766589s" totalImagesPullingTime="8.719306337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:48.201144258 +0000 UTC m=+11.872446876" watchObservedRunningTime="2026-11-03 16:01:48.201766589 +0000 UTC m=+11.873069199"
Nov 03 16:01:48.202309 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.202087    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hr4sq" podStartSLOduration=2.979133922 podStartE2EDuration="11.202065851s" totalImagesPullingTime="8.222931929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:48.174078137 +0000 UTC m=+11.845380744" watchObservedRunningTime="2026-11-03 16:01:48.202065851 +0000 UTC m=+11.873368461"
Nov 03 16:01:48.906785 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.906755    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:48.907233 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:48.906888    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:48.907307 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.907284    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:48.907421 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:48.907400    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:48.907495 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:48.907470    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:48.907557 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:48.907538    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:49.077305 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.077267    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4" event={"ID":"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198","Type":"ContainerStarted","Data":"9c20c7f54ffa4c0f1cf40f23c1077c75eccc872425d812b68466d2f1fa56915f"}
Nov 03 16:01:49.079701 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.079654    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wblql" event={"ID":"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7","Type":"ContainerStarted","Data":"d2efbbbdf3efccf638300ae502beaf682c6cf1f203833f5332f5a9a84ac970b3"}
Nov 03 16:01:49.082859 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.082830    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="a7beef7c73935651aeb41c187755117f3904d566df7051dc05080eeb1cf7854a" exitCode=0
Nov 03 16:01:49.082990 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.082937    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerDied","Data":"a7beef7c73935651aeb41c187755117f3904d566df7051dc05080eeb1cf7854a"}
Nov 03 16:01:49.115849 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.115788    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4" podStartSLOduration=3.169329305 podStartE2EDuration="12.115769012s" totalImagesPullingTime="8.946439707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:49.115151736 +0000 UTC m=+12.786454343" watchObservedRunningTime="2026-11-03 16:01:49.115769012 +0000 UTC m=+12.787071623"
Nov 03 16:01:49.181168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.181047    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wblql" podStartSLOduration=3.858022604 podStartE2EDuration="12.181027807s" totalImagesPullingTime="8.323005203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:49.180469192 +0000 UTC m=+12.851771801" watchObservedRunningTime="2026-11-03 16:01:49.181027807 +0000 UTC m=+12.852330439"
Nov 03 16:01:49.866059 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.866022    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:49.866214 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.866187    2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:01:49.866679 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:49.866656    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:01:50.088665 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.088622    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="44ee9540d273aae06022255f849efa3cf94c1405a077d6db107c6b9971ad46ae" exitCode=0
Nov 03 16:01:50.089331 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.088703    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerDied","Data":"44ee9540d273aae06022255f849efa3cf94c1405a077d6db107c6b9971ad46ae"}
Nov 03 16:01:50.090680 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.090657    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ww48r" event={"ID":"b903185c-6b3b-45dd-aac2-caaa87d0b075","Type":"ContainerStarted","Data":"c8a41cc96baf3323fbb4d9e076f32f532fa787af86285505586aa95bcec6e4ff"}
Nov 03 16:01:50.139448 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.139372    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ww48r" podStartSLOduration=2.525066538 podStartE2EDuration="13.139351808s" totalImagesPullingTime="10.61428527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:50.139005759 +0000 UTC m=+13.810308368" watchObservedRunningTime="2026-11-03 16:01:50.139351808 +0000 UTC m=+13.810654423"
Nov 03 16:01:50.906807 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.906771    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:50.906994 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:50.906898    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:50.907300 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.907280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:50.907431 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:50.907409    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:50.907509 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:50.907495    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:50.907605 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:50.907581    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:52.909836 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:52.909630    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:52.910381 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:52.909630    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:52.910381 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:52.909926    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:52.910381 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:52.909674    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:52.910381 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:52.910097    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:52.910381 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:52.909983    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:53.100253 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:53.100216    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="1e50f90380d10007cef412f977459a7d175945e1d413e90fea0f6d95d4094ad2" exitCode=0
Nov 03 16:01:53.100447 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:53.100272    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerDied","Data":"1e50f90380d10007cef412f977459a7d175945e1d413e90fea0f6d95d4094ad2"}
Nov 03 16:01:54.105526 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.105490    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="4eec39c758454315e809b597c758aa5bd8a521b17e3d60e4e50a0eae40cd66e5" exitCode=0
Nov 03 16:01:54.105921 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.105535    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerDied","Data":"4eec39c758454315e809b597c758aa5bd8a521b17e3d60e4e50a0eae40cd66e5"}
Nov 03 16:01:54.690711 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.690672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:54.690839 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.690714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:54.690839 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.690761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:54.690839 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690793    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:54.690839 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690828    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:01:54.691034 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690844    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:01:54.691034 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690843    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:54.691034 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690855    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8262d for pod openshift-network-diagnostics/network-check-target-kdkkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:54.691034 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690860    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.690841963 +0000 UTC m=+34.362144550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:01:54.691034 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690881    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret podName:d3387caf-55fb-4a86-9b89-f38c8fbe1915 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.690870819 +0000 UTC m=+34.362173436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret") pod "global-pull-secret-syncer-rtlt9" (UID: "d3387caf-55fb-4a86-9b89-f38c8fbe1915") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:01:54.691034 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.690894    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d podName:e3c566eb-b54b-47e7-a848-c1cacacc63e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.690888334 +0000 UTC m=+34.362190920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8262d" (UniqueName: "kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d") pod "network-check-target-kdkkw" (UID: "e3c566eb-b54b-47e7-a848-c1cacacc63e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:01:54.907087 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.906909    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:54.907271 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.906911    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:54.907271 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.907171    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:54.907271 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:54.906971    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:54.907457 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.907228    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:54.907457 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:54.907319    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:55.110319 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.110222    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"22638b1b32477d46828a4dba8363d9d2686c963d925639f7c2a939313b5c533e"}
Nov 03 16:01:55.110319 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.110296    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"d764caf5b1284095d7e4294ab6ce2dad185801629b06c5e9adf2ef78b924c9eb"}
Nov 03 16:01:55.110790 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.110333    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"8d1c0566359e86c688c0d813807cff96a0a2da2ab48b478bbab9709c9a4f49ea"}
Nov 03 16:01:55.110790 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.110364    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"6f3a345bb30bd4988e7d65dcce33483e2204f126dab07fe8bbf387ce7c8cc494"}
Nov 03 16:01:55.110790 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.110415    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"fb91f901c671e17ebc1f0acbcb84f22854cd7d8aa04136e7d8cdf9f245a94590"}
Nov 03 16:01:55.110790 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.110445    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"51e9576a50193ed2968f75a8ecc1e8130a68995aeaf66235c204424e68a2e4ee"}
Nov 03 16:01:55.113654 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.113622    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpddq" event={"ID":"fe4abe85-5418-426a-bb01-57b63c1f5c52","Type":"ContainerStarted","Data":"7fd1939d194a4d752805143f51d30dd21028eaa812d7dcde11f54aa361f61c5d"}
Nov 03 16:01:55.142919 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:55.142872    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zpddq" podStartSLOduration=7.29201004 podStartE2EDuration="18.142859669s" totalImagesPullingTime="10.850849629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:55.142632375 +0000 UTC m=+18.813934983" watchObservedRunningTime="2026-11-03 16:01:55.142859669 +0000 UTC m=+18.814162277"
Nov 03 16:01:56.907294 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:56.907254    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:56.908085 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:56.907364    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:56.908085 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:56.907372    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:56.908085 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:56.907415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:56.908085 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:56.907522    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:56.908085 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:56.907609    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:57.120439 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:57.120341    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"f5f5fc7e2b98e42722039fd2be8ff79668b1fd4d9cca41fab1e11643b4108c6a"}
Nov 03 16:01:58.906690 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:58.906656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:01:58.907081 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:58.906656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:01:58.907081 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:58.906771    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:01:58.907081 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:58.906851    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:01:58.907081 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:58.906665    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:01:58.907081 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:01:58.906962    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:01:59.128661 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.128617    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" event={"ID":"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc","Type":"ContainerStarted","Data":"8682838127eb2a2fc36d24b3b23b02bf334d5b01bd6fcb1641076a54d9a41b2c"}
Nov 03 16:01:59.128945 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.128892    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:59.129016 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.128965    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:59.129016 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.129005    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:59.144608 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.144582    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:59.144755 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.144688    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:01:59.165446 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:01:59.165318    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd" podStartSLOduration=6.937589916 podStartE2EDuration="22.165305715s" totalImagesPullingTime="15.227715799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:59.165231534 +0000 UTC m=+22.836534142" watchObservedRunningTime="2026-11-03 16:01:59.165305715 +0000 UTC m=+22.836608322"
Nov 03 16:02:00.677930 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:00.677900    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rtlt9"]
Nov 03 16:02:00.678670 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:00.678051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:00.678670 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:00.678138    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:02:00.680958 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:00.680928    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kdkkw"]
Nov 03 16:02:00.681100 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:00.681068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:00.681196 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:00.681177    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:02:00.684659 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:00.684632    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-njvgc"]
Nov 03 16:02:00.684801 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:00.684756    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:00.684865 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:00.684848    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:02:01.906619 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:01.906580    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:01.907081 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:01.906716    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:02:01.907174 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:01.907127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:01.907248 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:01.907223    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:02:02.906463 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:02.906424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:02.906668 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:02.906556    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:02:03.906498 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:03.906414    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:03.906647 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:03.906414    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:03.906647 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:03.906527    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:02:03.906647 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:03.906593    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:02:04.905983 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:04.905943    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:04.906181 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:04.906064    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:02:05.906037 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:05.906006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:05.906037 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:05.906045    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:05.906469 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:05.906129    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njvgc" podUID="1c0598b6-8a65-4e35-9347-907aa42e8781"
Nov 03 16:02:05.906469 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:05.906243    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtlt9" podUID="d3387caf-55fb-4a86-9b89-f38c8fbe1915"
Nov 03 16:02:06.907522 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:06.907490    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:06.907970 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:06.907581    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdkkw" podUID="e3c566eb-b54b-47e7-a848-c1cacacc63e7"
Nov 03 16:02:07.145154 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.145119    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeReady"
Nov 03 16:02:07.145340 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.145297    2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 16:02:07.197275 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.197240    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xw2n8"]
Nov 03 16:02:07.201827 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.201129    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vsbds"]
Nov 03 16:02:07.202064 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.202037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:07.204674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.204629    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wlfxh"]
Nov 03 16:02:07.204801 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.204726    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:07.208345 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.208325    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b94f5bc97-w9xjm"]
Nov 03 16:02:07.208477 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.208460    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:07.211167 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.211148    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"]
Nov 03 16:02:07.211271 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.211253    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.214069 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.214051    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.214166 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.214125    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.214563 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.214545    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vrzhj"]
Nov 03 16:02:07.214641 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.214626    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.218006 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.217986    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"]
Nov 03 16:02:07.218144 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.218125    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.222116 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.222095    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-859db4d5f7-xhncb"]
Nov 03 16:02:07.224957 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.223281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.224957 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.224178    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbm9f\""
Nov 03 16:02:07.224957 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.224244    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 16:02:07.224957 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.224245    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 16:02:07.224957 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.224298    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 16:02:07.225324 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.225306    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"]
Nov 03 16:02:07.225629 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.225610    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 16:02:07.225787 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.225769    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.228847 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.228824    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"]
Nov 03 16:02:07.230108 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.230086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.232409 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.231284    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"]
Nov 03 16:02:07.232409 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.231506    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.241883 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.238656    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d76ff499b-x7d7l"]
Nov 03 16:02:07.241883 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.240288    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.243176 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.243148    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"]
Nov 03 16:02:07.243870 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.243850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.249608 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.249585    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"]
Nov 03 16:02:07.249731 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.249689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:07.254486 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.254445    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"]
Nov 03 16:02:07.254648 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.254613    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.257844 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257822    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wlfxh"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257857    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xw2n8"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257878    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vsbds"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257895    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257910    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vrzhj"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257922    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257940    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"]
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:07.257973 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.257961    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b4wvw"]
Nov 03 16:02:07.261784 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.261765    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"]
Nov 03 16:02:07.261924 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.261803    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8qq6k"]
Nov 03 16:02:07.261924 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.261883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.264371 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.264352    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"]
Nov 03 16:02:07.264516 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.264381    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"]
Nov 03 16:02:07.264516 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.264432    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"]
Nov 03 16:02:07.264600 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.264540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.273784 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.273758    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4wvw"]
Nov 03 16:02:07.273913 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.273865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a6ba52-20d8-40fe-9723-54418819a3ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.273913 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.273898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngjp\" (UniqueName: \"kubernetes.io/projected/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2-kube-api-access-kngjp\") pod \"kas-connection-checker-685895c7c9-wlfxh\" (UID: \"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2\") " pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:07.274010 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.273928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274010 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.273998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274137 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274191 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkzkr\" (UniqueName: \"kubernetes.io/projected/315b5fc8-0084-4395-96a5-8ae4e210b974-kube-api-access-vkzkr\") pod \"kas-connection-checker-685895c7c9-xw2n8\" (UID: \"315b5fc8-0084-4395-96a5-8ae4e210b974\") " pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:07.274241 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274198    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274241 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.274338 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274338 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274269    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274338 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274291    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.274338 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274313    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dp9\" (UniqueName: \"kubernetes.io/projected/d7a6ba52-20d8-40fe-9723-54418819a3ab-kube-api-access-w5dp9\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.274547 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274339    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jl7\" (UniqueName: \"kubernetes.io/projected/dfe05141-01c7-4446-a5b5-e61abe65dfa2-kube-api-access-w8jl7\") pod \"kas-connection-checker-685895c7c9-vsbds\" (UID: \"dfe05141-01c7-4446-a5b5-e61abe65dfa2\") " pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:07.274547 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.274366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.286689 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.286661    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-859db4d5f7-xhncb"]
Nov 03 16:02:07.289572 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.289544    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8qq6k"]
Nov 03 16:02:07.294355 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.294326    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b94f5bc97-w9xjm"]
Nov 03 16:02:07.307874 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.307850    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d76ff499b-x7d7l"]
Nov 03 16:02:07.312532 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.312506    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"]
Nov 03 16:02:07.314832 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.314809    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 16:02:07.314953 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.314816    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315149    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.315278 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315167    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315278 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315225    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 16:02:07.315428 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315305    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.315428 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315342    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315428 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315375    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315712 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315695    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315775 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315721    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.315830 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315786    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315903 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315889    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315958 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315921    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.315958 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315950    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 16:02:07.316053 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316026    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.316053 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.315892    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.316137 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316078    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.316137 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316118    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 16:02:07.316457 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316437    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 16:02:07.316457 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316442    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.316568 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316503    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.316568 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316512    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.316629 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316570    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 16:02:07.316728 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316710    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.316826 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.316770    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 16:02:07.318237 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.318217    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 16:02:07.318862 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.318833    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 16:02:07.320252 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.320100    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 16:02:07.329451 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.328149    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 16:02:07.329451 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.328249    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 16:02:07.329451 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.328884    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 16:02:07.329451 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329102    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 16:02:07.329451 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329373    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 16:02:07.329451 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329451    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329483    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329515    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329540    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-ccc6l\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329606    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329618    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6rcnt\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329656    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329665    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329694    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 16:02:07.329825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329758    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 16:02:07.330187 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329853    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5c5gf\""
Nov 03 16:02:07.330187 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329881    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8szjr\""
Nov 03 16:02:07.330187 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329932    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jk68z\""
Nov 03 16:02:07.330187 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.329965    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 16:02:07.332575 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.332551    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 16:02:07.374866 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.374835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-tmp\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.374866 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.374873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.375107 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.374909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.375107 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.374999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-snapshots\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.375107 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375061    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.375225 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375145    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e92347-295e-4c8e-bef8-519fa58d4e87-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.375225 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4zf\" (UniqueName: \"kubernetes.io/projected/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-kube-api-access-jq4zf\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.375225 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkzkr\" (UniqueName: \"kubernetes.io/projected/315b5fc8-0084-4395-96a5-8ae4e210b974-kube-api-access-vkzkr\") pod \"kas-connection-checker-685895c7c9-xw2n8\" (UID: \"315b5fc8-0084-4395-96a5-8ae4e210b974\") " pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:07.375364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2v2p\" (UniqueName: \"kubernetes.io/projected/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-kube-api-access-b2v2p\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.375364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhp2\" (UniqueName: \"kubernetes.io/projected/d327c29c-36d3-4d01-8cab-bf10a76dbf48-kube-api-access-kvhp2\") pod \"volume-data-source-validator-58c9f5f46d-qfwp4\" (UID: \"d327c29c-36d3-4d01-8cab-bf10a76dbf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:07.375364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.375364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375315    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7tb\" (UniqueName: \"kubernetes.io/projected/2351ca8a-ffba-4733-8413-2ae0b585b489-kube-api-access-6x7tb\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.375364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375343    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.375613 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-default-certificate\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.375613 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375522    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.375613 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375574    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qbn\" (UniqueName: \"kubernetes.io/projected/a2e92347-295e-4c8e-bef8-519fa58d4e87-kube-api-access-n8qbn\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.375765 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375609    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.375765 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375642    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw48k\" (UniqueName: \"kubernetes.io/projected/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-kube-api-access-rw48k\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.375765 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375668    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-stats-auth\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.375765 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jl7\" (UniqueName: \"kubernetes.io/projected/dfe05141-01c7-4446-a5b5-e61abe65dfa2-kube-api-access-w8jl7\") pod \"kas-connection-checker-685895c7c9-vsbds\" (UID: \"dfe05141-01c7-4446-a5b5-e61abe65dfa2\") " pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:07.375959 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376012 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.375976    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.376074 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.376121 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.376121 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7g4\" (UniqueName: \"kubernetes.io/projected/22b39777-54ae-4a95-91bb-81943254f1aa-kube-api-access-rr7g4\") pod \"network-check-source-6f78d59f99-mhc9q\" (UID: \"22b39777-54ae-4a95-91bb-81943254f1aa\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:07.376219 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-tmp-dir\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.376219 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.376219 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376194    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376329 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376329 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376329 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.376487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-config\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.376487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376378    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-serving-cert\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.376487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.376487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj857\" (UniqueName: \"kubernetes.io/projected/e3dbcf1c-4684-46e8-b806-1df38410a5c9-kube-api-access-nj857\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.376640 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376534    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376640 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.376640 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376578    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e92347-295e-4c8e-bef8-519fa58d4e87-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.376640 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376586    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376640 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-config\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.376640 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376634    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vvlj\" (UniqueName: \"kubernetes.io/projected/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-kube-api-access-6vvlj\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dbcf1c-4684-46e8-b806-1df38410a5c9-cert\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-trusted-ca\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b592ced-7c65-40a8-94d7-cad2c29fe081-serving-cert\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.376925 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.376925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.377156 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.377156 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.377094    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:07.377156 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab65c33-a23d-48f3-ba16-212af80c62cb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.377189    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.877166129 +0000 UTC m=+31.548468716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.377206    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.377220    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b94f5bc97-w9xjm: secret "image-registry-tls" not found
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dp9\" (UniqueName: \"kubernetes.io/projected/d7a6ba52-20d8-40fe-9723-54418819a3ab-kube-api-access-w5dp9\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.377259    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls podName:17eca28e-8ea1-4193-92c1-32dd16b78e0d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.87724419 +0000 UTC m=+31.548546799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls") pod "image-registry-7b94f5bc97-w9xjm" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d") : secret "image-registry-tls" not found
Nov 03 16:02:07.377296 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377287    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a6ba52-20d8-40fe-9723-54418819a3ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.377622 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-config-volume\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.377622 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.377622 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377399    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngjp\" (UniqueName: \"kubernetes.io/projected/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2-kube-api-access-kngjp\") pod \"kas-connection-checker-685895c7c9-wlfxh\" (UID: \"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2\") " pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:07.377622 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377426    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5lp\" (UniqueName: \"kubernetes.io/projected/1b592ced-7c65-40a8-94d7-cad2c29fe081-kube-api-access-cr5lp\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.377622 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-serving-cert\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.377923 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.377906    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a6ba52-20d8-40fe-9723-54418819a3ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.379587 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.379563    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.379674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.379602    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.388039 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.387997    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkzkr\" (UniqueName: \"kubernetes.io/projected/315b5fc8-0084-4395-96a5-8ae4e210b974-kube-api-access-vkzkr\") pod \"kas-connection-checker-685895c7c9-xw2n8\" (UID: \"315b5fc8-0084-4395-96a5-8ae4e210b974\") " pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:07.388232 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.388066    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jl7\" (UniqueName: \"kubernetes.io/projected/dfe05141-01c7-4446-a5b5-e61abe65dfa2-kube-api-access-w8jl7\") pod \"kas-connection-checker-685895c7c9-vsbds\" (UID: \"dfe05141-01c7-4446-a5b5-e61abe65dfa2\") " pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:07.388232 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.388190    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.395674 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.395645    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngjp\" (UniqueName: \"kubernetes.io/projected/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2-kube-api-access-kngjp\") pod \"kas-connection-checker-685895c7c9-wlfxh\" (UID: \"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2\") " pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:07.395907 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.395885    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.396418 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.396381    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dp9\" (UniqueName: \"kubernetes.io/projected/d7a6ba52-20d8-40fe-9723-54418819a3ab-kube-api-access-w5dp9\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.478329 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvlj\" (UniqueName: \"kubernetes.io/projected/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-kube-api-access-6vvlj\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.478329 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dbcf1c-4684-46e8-b806-1df38410a5c9-cert\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478345    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-trusted-ca\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478452    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b592ced-7c65-40a8-94d7-cad2c29fe081-serving-cert\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.478596 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.478556    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:07.478930 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.478624    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.978604493 +0000 UTC m=+31.649907082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:07.478930 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab65c33-a23d-48f3-ba16-212af80c62cb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.479043 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-config-volume\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.479043 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.478990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.479043 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5lp\" (UniqueName: \"kubernetes.io/projected/1b592ced-7c65-40a8-94d7-cad2c29fe081-kube-api-access-cr5lp\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.479181 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-serving-cert\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.479181 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-tmp\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.479181 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479103    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.479181 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-snapshots\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.479181 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479144    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.479181 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479172    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479201    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e92347-295e-4c8e-bef8-519fa58d4e87-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479230    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4zf\" (UniqueName: \"kubernetes.io/projected/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-kube-api-access-jq4zf\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2v2p\" (UniqueName: \"kubernetes.io/projected/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-kube-api-access-b2v2p\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhp2\" (UniqueName: \"kubernetes.io/projected/d327c29c-36d3-4d01-8cab-bf10a76dbf48-kube-api-access-kvhp2\") pod \"volume-data-source-validator-58c9f5f46d-qfwp4\" (UID: \"d327c29c-36d3-4d01-8cab-bf10a76dbf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7tb\" (UniqueName: \"kubernetes.io/projected/2351ca8a-ffba-4733-8413-2ae0b585b489-kube-api-access-6x7tb\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479326    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-trusted-ca\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.479487 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.479423    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:02:07.479803 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.479516    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.979492161 +0000 UTC m=+31.650794782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : secret "router-metrics-certs-default" not found
Nov 03 16:02:07.479803 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.479641    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.480042 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480021    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.480175 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480149    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab65c33-a23d-48f3-ba16-212af80c62cb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.480227 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480183    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-config-volume\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.480529 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.480619 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-default-certificate\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.480619 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480552    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-tmp\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.480619 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qbn\" (UniqueName: \"kubernetes.io/projected/a2e92347-295e-4c8e-bef8-519fa58d4e87-kube-api-access-n8qbn\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.480619 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw48k\" (UniqueName: \"kubernetes.io/projected/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-kube-api-access-rw48k\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-stats-auth\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480756    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7g4\" (UniqueName: \"kubernetes.io/projected/22b39777-54ae-4a95-91bb-81943254f1aa-kube-api-access-rr7g4\") pod \"network-check-source-6f78d59f99-mhc9q\" (UID: \"22b39777-54ae-4a95-91bb-81943254f1aa\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:07.480821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480813    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-tmp-dir\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-config\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-serving-cert\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.480986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj857\" (UniqueName: \"kubernetes.io/projected/e3dbcf1c-4684-46e8-b806-1df38410a5c9-kube-api-access-nj857\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e92347-295e-4c8e-bef8-519fa58d4e87-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.481152 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-config\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.481622 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481602    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.481682 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481650    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dbcf1c-4684-46e8-b806-1df38410a5c9-cert\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.481735 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481715    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b592ced-7c65-40a8-94d7-cad2c29fe081-serving-cert\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.481789 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481750    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-config\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.481993    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-tmp-dir\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482000    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.981981983 +0000 UTC m=+31.653284572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : configmap references non-existent config key: service-ca.crt
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.482106    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.482187    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-config\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482202    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482218    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d76ff499b-x7d7l: secret "image-registry-tls" not found
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482274    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls podName:df53f549-e19a-4304-8387-4afa8528b975 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.982258133 +0000 UTC m=+31.653560729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls") pod "image-registry-6d76ff499b-x7d7l" (UID: "df53f549-e19a-4304-8387-4afa8528b975") : secret "image-registry-tls" not found
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482279    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482338    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482343    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls podName:66c8687d-b9da-46b2-85ce-52cd8f6ac1dd nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.982308786 +0000 UTC m=+31.653611380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls") pod "dns-default-b4wvw" (UID: "66c8687d-b9da-46b2-85ce-52cd8f6ac1dd") : secret "dns-default-metrics-tls" not found
Nov 03 16:02:07.482413 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.482369    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls podName:da622b12-25ac-43dc-8fbb-c9a974aa9f9d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.98235992 +0000 UTC m=+31.653662520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vq8cr" (UID: "da622b12-25ac-43dc-8fbb-c9a974aa9f9d") : secret "samples-operator-tls" not found
Nov 03 16:02:07.483012 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.482523    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-snapshots\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.483012 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.482744    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-serving-cert\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.483012 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.482861    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.483012 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.482884    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e92347-295e-4c8e-bef8-519fa58d4e87-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.484471 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.484119    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-serving-cert\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.484471 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.484562    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e92347-295e-4c8e-bef8-519fa58d4e87-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.484774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.484758    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.484831 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.484790    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-default-certificate\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.485539 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.485513    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-stats-auth\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.486066 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.486050    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.497234 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.497208    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvlj\" (UniqueName: \"kubernetes.io/projected/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-kube-api-access-6vvlj\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.497340 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.497284    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5lp\" (UniqueName: \"kubernetes.io/projected/1b592ced-7c65-40a8-94d7-cad2c29fe081-kube-api-access-cr5lp\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.497744 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.497728    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7g4\" (UniqueName: \"kubernetes.io/projected/22b39777-54ae-4a95-91bb-81943254f1aa-kube-api-access-rr7g4\") pod \"network-check-source-6f78d59f99-mhc9q\" (UID: \"22b39777-54ae-4a95-91bb-81943254f1aa\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:07.500821 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.500797    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qbn\" (UniqueName: \"kubernetes.io/projected/a2e92347-295e-4c8e-bef8-519fa58d4e87-kube-api-access-n8qbn\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.500954 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.500889    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2v2p\" (UniqueName: \"kubernetes.io/projected/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-kube-api-access-b2v2p\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.501355 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.501330    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7tb\" (UniqueName: \"kubernetes.io/projected/2351ca8a-ffba-4733-8413-2ae0b585b489-kube-api-access-6x7tb\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.501644 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.501621    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhp2\" (UniqueName: \"kubernetes.io/projected/d327c29c-36d3-4d01-8cab-bf10a76dbf48-kube-api-access-kvhp2\") pod \"volume-data-source-validator-58c9f5f46d-qfwp4\" (UID: \"d327c29c-36d3-4d01-8cab-bf10a76dbf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:07.501883 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.501861    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.502093 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.502069    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj857\" (UniqueName: \"kubernetes.io/projected/e3dbcf1c-4684-46e8-b806-1df38410a5c9-kube-api-access-nj857\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.502509 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.502489    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw48k\" (UniqueName: \"kubernetes.io/projected/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-kube-api-access-rw48k\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.502571 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.502549    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4zf\" (UniqueName: \"kubernetes.io/projected/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-kube-api-access-jq4zf\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.503150 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.503133    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.515500 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.515466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:07.522425 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.522399    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:07.533647 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.533617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:07.562862 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.562831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:07.567879 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.567846    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:07.591424 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.591360    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:07.609424 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.608592    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:07.620370 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.615264    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:07.637921 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.637886    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:07.645174 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.645108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:07.834870 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.834423    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xw2n8"]
Nov 03 16:02:07.837862 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.837515    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wlfxh"]
Nov 03 16:02:07.837951 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.837918    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315b5fc8_0084_4395_96a5_8ae4e210b974.slice/crio-3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513 WatchSource:0}: Error finding container 3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513: Status 404 returned error can't find the container with id 3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513
Nov 03 16:02:07.838892 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.838848    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vsbds"]
Nov 03 16:02:07.840335 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.840308    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc94f4e0_d6f9_4b11_8495_36cdf02aa8a2.slice/crio-e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54 WatchSource:0}: Error finding container e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54: Status 404 returned error can't find the container with id e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54
Nov 03 16:02:07.841072 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.841024    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe05141_01c7_4446_a5b5_e61abe65dfa2.slice/crio-bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82 WatchSource:0}: Error finding container bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82: Status 404 returned error can't find the container with id bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82
Nov 03 16:02:07.885368 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.885327    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:07.885556 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.885409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:07.885621 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.885576    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:07.885621 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.885594    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b94f5bc97-w9xjm: secret "image-registry-tls" not found
Nov 03 16:02:07.885621 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.885598    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:07.885742 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.885664    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls podName:17eca28e-8ea1-4193-92c1-32dd16b78e0d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.885643491 +0000 UTC m=+32.556946096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls") pod "image-registry-7b94f5bc97-w9xjm" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d") : secret "image-registry-tls" not found
Nov 03 16:02:07.885742 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.885683    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.885673394 +0000 UTC m=+32.556975982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:07.901965 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.900888    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"]
Nov 03 16:02:07.908614 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.906595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:07.908614 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.907095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:07.912774 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.910118    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b592ced_7c65_40a8_94d7_cad2c29fe081.slice/crio-a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376 WatchSource:0}: Error finding container a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376: Status 404 returned error can't find the container with id a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376
Nov 03 16:02:07.912774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.911347    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:02:07.912774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.911604    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\""
Nov 03 16:02:07.912774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.911667    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\""
Nov 03 16:02:07.912774 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.912764    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:02:07.919565 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.916136    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vrzhj"]
Nov 03 16:02:07.919565 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.916270    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4f4939_d15b_4b06_bc6e_27ca8b2039ec.slice/crio-ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8 WatchSource:0}: Error finding container ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8: Status 404 returned error can't find the container with id ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8
Nov 03 16:02:07.940642 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.940489    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"]
Nov 03 16:02:07.949178 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.948652    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e92347_295e_4c8e_bef8_519fa58d4e87.slice/crio-d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356 WatchSource:0}: Error finding container d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356: Status 404 returned error can't find the container with id d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356
Nov 03 16:02:07.950291 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.950267    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"]
Nov 03 16:02:07.957373 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.957350    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71be33d_a6e5_4b3a_b703_64cc2f214f5b.slice/crio-3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52 WatchSource:0}: Error finding container 3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52: Status 404 returned error can't find the container with id 3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52
Nov 03 16:02:07.988412 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.982602    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"]
Nov 03 16:02:07.990840 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:07.990799    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd327c29c_36d3_4d01_8cab_bf10a76dbf48.slice/crio-cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6 WatchSource:0}: Error finding container cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6: Status 404 returned error can't find the container with id cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6
Nov 03 16:02:07.990976 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.990872    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.990976 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.990941    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:07.991101 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.990981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:07.991101 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.991009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:07.991101 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.991058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:07.991235 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:07.991101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:07.991290 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991275    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991335    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.991317748 +0000 UTC m=+32.662620347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991501    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991514    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991534    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d76ff499b-x7d7l: secret "image-registry-tls" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991560    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.991544199 +0000 UTC m=+32.662846803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : secret "router-metrics-certs-default" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991581    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls podName:df53f549-e19a-4304-8387-4afa8528b975 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.991571615 +0000 UTC m=+32.662874205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls") pod "image-registry-6d76ff499b-x7d7l" (UID: "df53f549-e19a-4304-8387-4afa8528b975") : secret "image-registry-tls" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991631    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.991613752 +0000 UTC m=+32.662916339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : configmap references non-existent config key: service-ca.crt
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991639    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991671    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls podName:da622b12-25ac-43dc-8fbb-c9a974aa9f9d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.991661336 +0000 UTC m=+32.662963952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vq8cr" (UID: "da622b12-25ac-43dc-8fbb-c9a974aa9f9d") : secret "samples-operator-tls" not found
Nov 03 16:02:07.991700 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991697    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:02:07.992236 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:07.991730    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls podName:66c8687d-b9da-46b2-85ce-52cd8f6ac1dd nodeName:}" failed. No retries permitted until 2026-11-03 16:02:08.991719213 +0000 UTC m=+32.663021805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls") pod "dns-default-b4wvw" (UID: "66c8687d-b9da-46b2-85ce-52cd8f6ac1dd") : secret "dns-default-metrics-tls" not found
Nov 03 16:02:08.012011 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.011894    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"]
Nov 03 16:02:08.016017 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:08.015987    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b39777_54ae_4a95_91bb_81943254f1aa.slice/crio-d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e WatchSource:0}: Error finding container d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e: Status 404 returned error can't find the container with id d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e
Nov 03 16:02:08.021351 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.021329    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8qq6k"]
Nov 03 16:02:08.024237 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:08.024212    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dbcf1c_4684_46e8_b806_1df38410a5c9.slice/crio-eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1 WatchSource:0}: Error finding container eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1: Status 404 returned error can't find the container with id eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1
Nov 03 16:02:08.148873 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.148839    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q" event={"ID":"22b39777-54ae-4a95-91bb-81943254f1aa","Type":"ContainerStarted","Data":"d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e"}
Nov 03 16:02:08.149937 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.149912    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q" event={"ID":"a2e92347-295e-4c8e-bef8-519fa58d4e87","Type":"ContainerStarted","Data":"d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356"}
Nov 03 16:02:08.151300 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.151278    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vsbds" event={"ID":"dfe05141-01c7-4446-a5b5-e61abe65dfa2","Type":"ContainerStarted","Data":"cb77f01a7f40d6aa0005ce04000eeea57a135b75259ed0954794e161396efa90"}
Nov 03 16:02:08.151404 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.151312    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vsbds" event={"ID":"dfe05141-01c7-4446-a5b5-e61abe65dfa2","Type":"ContainerStarted","Data":"bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82"}
Nov 03 16:02:08.152516 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.152498    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4" event={"ID":"d327c29c-36d3-4d01-8cab-bf10a76dbf48","Type":"ContainerStarted","Data":"cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6"}
Nov 03 16:02:08.153825 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.153801    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wlfxh" event={"ID":"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2","Type":"ContainerStarted","Data":"c58e1cb08127c37a21106240df83613ab4df9703f52ec5c7a3f25ff97c31bcff"}
Nov 03 16:02:08.153917 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.153840    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wlfxh" event={"ID":"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2","Type":"ContainerStarted","Data":"e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54"}
Nov 03 16:02:08.154853 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.154833    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8qq6k" event={"ID":"e3dbcf1c-4684-46e8-b806-1df38410a5c9","Type":"ContainerStarted","Data":"eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1"}
Nov 03 16:02:08.155909 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.155886    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb" event={"ID":"a71be33d-a6e5-4b3a-b703-64cc2f214f5b","Type":"ContainerStarted","Data":"3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52"}
Nov 03 16:02:08.156922 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.156905    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" event={"ID":"1b592ced-7c65-40a8-94d7-cad2c29fe081","Type":"ContainerStarted","Data":"a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376"}
Nov 03 16:02:08.158140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.158120    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xw2n8" event={"ID":"315b5fc8-0084-4395-96a5-8ae4e210b974","Type":"ContainerStarted","Data":"c113b36908e55f656ab1c95ec8c4641260fd59e570ba221bec206f2f23aa7c2d"}
Nov 03 16:02:08.158214 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.158158    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xw2n8" event={"ID":"315b5fc8-0084-4395-96a5-8ae4e210b974","Type":"ContainerStarted","Data":"3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513"}
Nov 03 16:02:08.159114 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.159098    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vrzhj" event={"ID":"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec","Type":"ContainerStarted","Data":"ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8"}
Nov 03 16:02:08.172819 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.172770    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vsbds" podStartSLOduration=335.172757345 podStartE2EDuration="5m35.172757345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:08.172250513 +0000 UTC m=+31.843553123" watchObservedRunningTime="2026-11-03 16:02:08.172757345 +0000 UTC m=+31.844059952"
Nov 03 16:02:08.196193 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.196143    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xw2n8" podStartSLOduration=335.196127683 podStartE2EDuration="5m35.196127683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:08.195311462 +0000 UTC m=+31.866614092" watchObservedRunningTime="2026-11-03 16:02:08.196127683 +0000 UTC m=+31.867430290"
Nov 03 16:02:08.214559 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.214501    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wlfxh" podStartSLOduration=335.214485631 podStartE2EDuration="5m35.214485631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:08.213580625 +0000 UTC m=+31.884883246" watchObservedRunningTime="2026-11-03 16:02:08.214485631 +0000 UTC m=+31.885788301"
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.900157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.900220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:08.900438    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:08.900454    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b94f5bc97-w9xjm: secret "image-registry-tls" not found
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:08.900526    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls podName:17eca28e-8ea1-4193-92c1-32dd16b78e0d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.900504281 +0000 UTC m=+34.571806882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls") pod "image-registry-7b94f5bc97-w9xjm" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d") : secret "image-registry-tls" not found
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:08.900957    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:08.901047 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:08.901006    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.900991172 +0000 UTC m=+34.572293772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:08.911777 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.911725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:08.916787 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:08.916745    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\""
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:09.000897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:09.000963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:09.001062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:09.001127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:09.001228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.001463    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:11.001441149 +0000 UTC m=+34.672743742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : configmap references non-existent config key: service-ca.crt
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.001971    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.002051    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls podName:da622b12-25ac-43dc-8fbb-c9a974aa9f9d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:11.002009578 +0000 UTC m=+34.673312165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vq8cr" (UID: "da622b12-25ac-43dc-8fbb-c9a974aa9f9d") : secret "samples-operator-tls" not found
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.002146    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.002186    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls podName:66c8687d-b9da-46b2-85ce-52cd8f6ac1dd nodeName:}" failed. No retries permitted until 2026-11-03 16:02:11.002174306 +0000 UTC m=+34.673476894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls") pod "dns-default-b4wvw" (UID: "66c8687d-b9da-46b2-85ce-52cd8f6ac1dd") : secret "dns-default-metrics-tls" not found
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.002254    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.002265    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d76ff499b-x7d7l: secret "image-registry-tls" not found
Nov 03 16:02:09.002380 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.002356    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls podName:df53f549-e19a-4304-8387-4afa8528b975 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:11.002323381 +0000 UTC m=+34.673625969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls") pod "image-registry-6d76ff499b-x7d7l" (UID: "df53f549-e19a-4304-8387-4afa8528b975") : secret "image-registry-tls" not found
Nov 03 16:02:09.003499 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:09.003456    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:09.003637 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.003622    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:02:09.003705 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.003673    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:11.00365979 +0000 UTC m=+34.674962381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : secret "router-metrics-certs-default" not found
Nov 03 16:02:09.003870 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.003736    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:09.003870 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:09.003769    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:11.003758553 +0000 UTC m=+34.675061141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:10.723860 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.723769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:10.724364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.723895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:10.724364 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.723939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:10.724364 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.724073    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 16:02:10.724364 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.724154    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:42.724131084 +0000 UTC m=+66.395433692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : secret "metrics-daemon-secret" not found
Nov 03 16:02:10.743300 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.743266    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:10.755644 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.755618    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:10.926094 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.926053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:10.926294 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:10.926103    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:10.926294 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.926272    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:10.926294 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.926287    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b94f5bc97-w9xjm: secret "image-registry-tls" not found
Nov 03 16:02:10.926497 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.926346    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls podName:17eca28e-8ea1-4193-92c1-32dd16b78e0d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:14.926326958 +0000 UTC m=+38.597629550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls") pod "image-registry-7b94f5bc97-w9xjm" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d") : secret "image-registry-tls" not found
Nov 03 16:02:10.926817 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.926790    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:10.926926 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:10.926853    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:14.926837535 +0000 UTC m=+38.598140121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:11.008245 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.008163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:11.026997 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.026912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:11.027076 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.027018    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:11.027132 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.027085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:11.027132 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.027126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:11.027244 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.027154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:11.027244 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.027196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:11.027374 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027339    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:11.027374 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027357    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d76ff499b-x7d7l: secret "image-registry-tls" not found
Nov 03 16:02:11.027843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027544    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:02:11.027843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027604    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls podName:da622b12-25ac-43dc-8fbb-c9a974aa9f9d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:15.027585343 +0000 UTC m=+38.698887949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vq8cr" (UID: "da622b12-25ac-43dc-8fbb-c9a974aa9f9d") : secret "samples-operator-tls" not found
Nov 03 16:02:11.027843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027674    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:11.027843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027711    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:15.027699256 +0000 UTC m=+38.699001842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:11.027843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027762    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:02:11.027843 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.027789    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:15.027780506 +0000 UTC m=+38.699083097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : secret "router-metrics-certs-default" not found
Nov 03 16:02:11.028428 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.028205    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:15.028189074 +0000 UTC m=+38.699491664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : configmap references non-existent config key: service-ca.crt
Nov 03 16:02:11.028428 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.028282    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:02:11.028428 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.028317    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls podName:66c8687d-b9da-46b2-85ce-52cd8f6ac1dd nodeName:}" failed. No retries permitted until 2026-11-03 16:02:15.028306416 +0000 UTC m=+38.699609003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls") pod "dns-default-b4wvw" (UID: "66c8687d-b9da-46b2-85ce-52cd8f6ac1dd") : secret "dns-default-metrics-tls" not found
Nov 03 16:02:11.028428 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:11.028370    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls podName:df53f549-e19a-4304-8387-4afa8528b975 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:15.028355936 +0000 UTC m=+38.699658526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls") pod "image-registry-6d76ff499b-x7d7l" (UID: "df53f549-e19a-4304-8387-4afa8528b975") : secret "image-registry-tls" not found
Nov 03 16:02:11.052898 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.046249    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:11.173887 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.173035    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4" event={"ID":"d327c29c-36d3-4d01-8cab-bf10a76dbf48","Type":"ContainerStarted","Data":"e2ca0e41379a478c2fe2ec88b375d4487ab8bb44a71c3b35f3c8372871f13b18"}
Nov 03 16:02:11.337719 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.336749    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4" podStartSLOduration=305.915588494 podStartE2EDuration="5m8.336719382s" totalImagesPullingTime="2.421130888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:11.197460597 +0000 UTC m=+34.868763204" watchObservedRunningTime="2026-11-03 16:02:11.336719382 +0000 UTC m=+35.008021995"
Nov 03 16:02:11.337719 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.337589    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rtlt9"]
Nov 03 16:02:11.344509 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:11.344437    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3387caf_55fb_4a86_9b89_f38c8fbe1915.slice/crio-d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915 WatchSource:0}: Error finding container d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915: Status 404 returned error can't find the container with id d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915
Nov 03 16:02:11.363877 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:11.363842    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kdkkw"]
Nov 03 16:02:11.370463 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:11.370425    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c566eb_b54b_47e7_a848_c1cacacc63e7.slice/crio-6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb WatchSource:0}: Error finding container 6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb: Status 404 returned error can't find the container with id 6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb
Nov 03 16:02:12.178291 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:12.178249    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kdkkw" event={"ID":"e3c566eb-b54b-47e7-a848-c1cacacc63e7","Type":"ContainerStarted","Data":"6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb"}
Nov 03 16:02:12.184636 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:12.184605    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rtlt9" event={"ID":"d3387caf-55fb-4a86-9b89-f38c8fbe1915","Type":"ContainerStarted","Data":"d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915"}
Nov 03 16:02:13.088242 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.087876    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bcvq5"]
Nov 03 16:02:13.107150 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.107112    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-b4vvv-lcf5m"]
Nov 03 16:02:13.108285 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.107486    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.111694 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.111668    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 16:02:13.112964 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.112919    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 16:02:13.113140 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.113058    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bb2h\""
Nov 03 16:02:13.117861 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.117318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.122418 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.122378    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dfrrd\""
Nov 03 16:02:13.124645 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.124561    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bcvq5"]
Nov 03 16:02:13.127375 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.127306    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-b4vvv-lcf5m"]
Nov 03 16:02:13.195483 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.195429    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb" event={"ID":"a71be33d-a6e5-4b3a-b703-64cc2f214f5b","Type":"ContainerStarted","Data":"ccec6db0c1cf30f16ff022dc8d2d6b14a1712e9f690d6f040cc06b70804f823a"}
Nov 03 16:02:13.197837 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.197811    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/0.log"
Nov 03 16:02:13.197981 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.197854    2584 generic.go:356] "Generic (PLEG): container finished" podID="1b592ced-7c65-40a8-94d7-cad2c29fe081" containerID="df57eceb5ba32214e6b350b5c1304f9f93af2be77290221e89650864697a45f1" exitCode=255
Nov 03 16:02:13.198038 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.197969    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" event={"ID":"1b592ced-7c65-40a8-94d7-cad2c29fe081","Type":"ContainerDied","Data":"df57eceb5ba32214e6b350b5c1304f9f93af2be77290221e89650864697a45f1"}
Nov 03 16:02:13.198278 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.198258    2584 scope.go:122] "RemoveContainer" containerID="df57eceb5ba32214e6b350b5c1304f9f93af2be77290221e89650864697a45f1"
Nov 03 16:02:13.200543 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.200511    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vrzhj" event={"ID":"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec","Type":"ContainerStarted","Data":"2e6824e3e27dd9d188e0c6d8af180b09da37caebdeb1259041a092a7e8c88b77"}
Nov 03 16:02:13.216798 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.216733    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb" podStartSLOduration=305.376304912 podStartE2EDuration="5m10.216713275s" totalImagesPullingTime="4.840408363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:13.215443519 +0000 UTC m=+36.886746153" watchObservedRunningTime="2026-11-03 16:02:13.216713275 +0000 UTC m=+36.888015879"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248678    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4beaa779-098d-4007-88f8-25796f1758f0-data-volume\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248838    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5tr\" (UniqueName: \"kubernetes.io/projected/4beaa779-098d-4007-88f8-25796f1758f0-kube-api-access-kw5tr\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.249126 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.248943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4beaa779-098d-4007-88f8-25796f1758f0-crio-socket\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.268463 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.268398    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-vrzhj" podStartSLOduration=305.735527151 podStartE2EDuration="5m10.268361238s" totalImagesPullingTime="4.532834087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:13.26756315 +0000 UTC m=+36.938865783" watchObservedRunningTime="2026-11-03 16:02:13.268361238 +0000 UTC m=+36.939663846"
Nov 03 16:02:13.349627 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.349627 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.349835 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349652    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.349835 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349729    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.349835 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4beaa779-098d-4007-88f8-25796f1758f0-data-volume\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.349986 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.349986 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5tr\" (UniqueName: \"kubernetes.io/projected/4beaa779-098d-4007-88f8-25796f1758f0-kube-api-access-kw5tr\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.350086 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.349985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.350086 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.350050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4beaa779-098d-4007-88f8-25796f1758f0-crio-socket\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.351010 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.350986    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:13.351137 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.351057    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:13.851035737 +0000 UTC m=+37.522338336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:13.351331 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.351316    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:02:13.351400 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.351358    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:13.851346223 +0000 UTC m=+37.522648822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : configmap "kube-rbac-proxy" not found
Nov 03 16:02:13.352111 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.352086    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4beaa779-098d-4007-88f8-25796f1758f0-data-volume\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.352963 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.352934    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4beaa779-098d-4007-88f8-25796f1758f0-crio-socket\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.353429 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.353406    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.353747 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.353725    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.353873 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.353848    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.365339 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.365306    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.365514 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.365308    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5tr\" (UniqueName: \"kubernetes.io/projected/4beaa779-098d-4007-88f8-25796f1758f0-kube-api-access-kw5tr\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.491819 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.491245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:13.823229 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.823206    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-b4vvv-lcf5m"]
Nov 03 16:02:13.855455 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.855415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.855650 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:13.855538    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:13.855650 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.855590    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:13.855753 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.855652    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:02:13.855753 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.855666    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:14.855643539 +0000 UTC m=+38.526946126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:13.855753 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:13.855711    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:14.855696031 +0000 UTC m=+38.526998617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : configmap "kube-rbac-proxy" not found
Nov 03 16:02:13.878270 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:13.878188    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a088fef_29ab_4b9e_a90d_f6560aa4ee43.slice/crio-94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9 WatchSource:0}: Error finding container 94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9: Status 404 returned error can't find the container with id 94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9
Nov 03 16:02:14.211066 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.210978    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8qq6k" event={"ID":"e3dbcf1c-4684-46e8-b806-1df38410a5c9","Type":"ContainerStarted","Data":"0906bee6d537cba843268fbde7a50e60c808d73ef0344539f541411fee7698de"}
Nov 03 16:02:14.216351 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.216322    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/1.log"
Nov 03 16:02:14.220425 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.218836    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/0.log"
Nov 03 16:02:14.220425 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.218881    2584 generic.go:356] "Generic (PLEG): container finished" podID="1b592ced-7c65-40a8-94d7-cad2c29fe081" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91" exitCode=255
Nov 03 16:02:14.220425 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.219038    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" event={"ID":"1b592ced-7c65-40a8-94d7-cad2c29fe081","Type":"ContainerDied","Data":"d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91"}
Nov 03 16:02:14.220425 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.219094    2584 scope.go:122] "RemoveContainer" containerID="df57eceb5ba32214e6b350b5c1304f9f93af2be77290221e89650864697a45f1"
Nov 03 16:02:14.220425 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.219551    2584 scope.go:122] "RemoveContainer" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91"
Nov 03 16:02:14.220425 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.219780    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4ftv2_openshift-console-operator(1b592ced-7c65-40a8-94d7-cad2c29fe081)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" podUID="1b592ced-7c65-40a8-94d7-cad2c29fe081"
Nov 03 16:02:14.224137 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.223577    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q" event={"ID":"a2e92347-295e-4c8e-bef8-519fa58d4e87","Type":"ContainerStarted","Data":"ac9c321bf25bc75fa90a07216a4af7cbe966ab40daf0e5adfabf2d43fa45f07a"}
Nov 03 16:02:14.227227 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.227198    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m" event={"ID":"0a088fef-29ab-4b9e-a90d-f6560aa4ee43","Type":"ContainerStarted","Data":"9d84057fe6988cea327584b7c0daa252502b8c1cc8f1987fb3ca195db75eb9b0"}
Nov 03 16:02:14.227355 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.227252    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m" event={"ID":"0a088fef-29ab-4b9e-a90d-f6560aa4ee43","Type":"ContainerStarted","Data":"94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9"}
Nov 03 16:02:14.252343 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.252281    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vpqq9"]
Nov 03 16:02:14.266373 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.266316    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.280465 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.273855    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8qq6k" podStartSLOduration=1.849986049 podStartE2EDuration="7.273837965s" totalImagesPullingTime="5.423851916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:14.273224438 +0000 UTC m=+37.944527060" watchObservedRunningTime="2026-11-03 16:02:14.273837965 +0000 UTC m=+37.945140571"
Nov 03 16:02:14.280465 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.275410    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vpqq9"]
Nov 03 16:02:14.280465 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.279096    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 16:02:14.280465 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.279213    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f7qmg\""
Nov 03 16:02:14.300953 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.300921    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 16:02:14.301104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.300981    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 16:02:14.301104 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.301017    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 16:02:14.322055 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.321999    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m" podStartSLOduration=1.321984306 podStartE2EDuration="1.321984306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:14.32037695 +0000 UTC m=+37.991679557" watchObservedRunningTime="2026-11-03 16:02:14.321984306 +0000 UTC m=+37.993286913"
Nov 03 16:02:14.342487 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.341180    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q" podStartSLOduration=305.372620172 podStartE2EDuration="5m11.3411624s" totalImagesPullingTime="5.968542228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:14.339497179 +0000 UTC m=+38.010799791" watchObservedRunningTime="2026-11-03 16:02:14.3411624 +0000 UTC m=+38.012465025"
Nov 03 16:02:14.370404 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.362172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-key\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.370404 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.362251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-cabundle\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.370404 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.362616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxw4\" (UniqueName: \"kubernetes.io/projected/646f7db2-f965-4cfc-883c-646eaf2975e5-kube-api-access-2pxw4\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.463219 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.463142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-key\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.463219 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.463194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-cabundle\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.463395 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.463326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxw4\" (UniqueName: \"kubernetes.io/projected/646f7db2-f965-4cfc-883c-646eaf2975e5-kube-api-access-2pxw4\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.464096 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.464072    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-cabundle\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.476643 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.475408    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-key\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.476643 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.476552    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxw4\" (UniqueName: \"kubernetes.io/projected/646f7db2-f965-4cfc-883c-646eaf2975e5-kube-api-access-2pxw4\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.599693 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.599653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:14.867007 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.866921    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:14.867007 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.866966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:14.867182 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.867096    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:02:14.867182 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.867097    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:14.867182 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.867152    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:16.867132957 +0000 UTC m=+40.538435560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : configmap "kube-rbac-proxy" not found
Nov 03 16:02:14.867182 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.867168    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:16.86715985 +0000 UTC m=+40.538462436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:14.898671 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.898553    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vpqq9"]
Nov 03 16:02:14.902498 ip-10-0-133-129 kubenswrapper[2584]: W1103 16:02:14.902468    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646f7db2_f965_4cfc_883c_646eaf2975e5.slice/crio-3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac WatchSource:0}: Error finding container 3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac: Status 404 returned error can't find the container with id 3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac
Nov 03 16:02:14.968073 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.967879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:14.968073 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:14.967940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:14.968073 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.968014    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:14.968297 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.968085    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:22.968054596 +0000 UTC m=+46.639357185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:14.968297 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.968091    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:14.968297 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.968106    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b94f5bc97-w9xjm: secret "image-registry-tls" not found
Nov 03 16:02:14.968297 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:14.968178    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls podName:17eca28e-8ea1-4193-92c1-32dd16b78e0d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:22.968140422 +0000 UTC m=+46.639443026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls") pod "image-registry-7b94f5bc97-w9xjm" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d") : secret "image-registry-tls" not found
Nov 03 16:02:15.069503 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.069459    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:15.069686 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.069550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:15.069686 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.069589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:15.069686 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069612    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069697    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.069619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069689    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.069668494 +0000 UTC m=+46.740971094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : secret "router-metrics-certs-default" not found
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069696    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069748    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.069733036 +0000 UTC m=+46.741035623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : configmap references non-existent config key: service-ca.crt
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069766    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls podName:66c8687d-b9da-46b2-85ce-52cd8f6ac1dd nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.069756875 +0000 UTC m=+46.741059462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls") pod "dns-default-b4wvw" (UID: "66c8687d-b9da-46b2-85ce-52cd8f6ac1dd") : secret "dns-default-metrics-tls" not found
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069791    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls podName:da622b12-25ac-43dc-8fbb-c9a974aa9f9d nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.069782318 +0000 UTC m=+46.741084909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vq8cr" (UID: "da622b12-25ac-43dc-8fbb-c9a974aa9f9d") : secret "samples-operator-tls" not found
Nov 03 16:02:15.069841 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.069841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:15.070358 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.069883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:15.070358 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069957    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:02:15.070358 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.069976    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d76ff499b-x7d7l: secret "image-registry-tls" not found
Nov 03 16:02:15.070358 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.070016    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls podName:df53f549-e19a-4304-8387-4afa8528b975 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.070005794 +0000 UTC m=+46.741308381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls") pod "image-registry-6d76ff499b-x7d7l" (UID: "df53f549-e19a-4304-8387-4afa8528b975") : secret "image-registry-tls" not found
Nov 03 16:02:15.070358 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.070069    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:15.070358 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.070151    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.070126648 +0000 UTC m=+46.741429241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:15.233013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.232984    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/1.log"
Nov 03 16:02:15.233534 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.233407    2584 scope.go:122] "RemoveContainer" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91"
Nov 03 16:02:15.233651 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:15.233629    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4ftv2_openshift-console-operator(1b592ced-7c65-40a8-94d7-cad2c29fe081)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" podUID="1b592ced-7c65-40a8-94d7-cad2c29fe081"
Nov 03 16:02:15.234657 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.234635    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kdkkw" event={"ID":"e3c566eb-b54b-47e7-a848-c1cacacc63e7","Type":"ContainerStarted","Data":"f1e31be94fddebd1a8002c2acf4ca8724628c487cf478dc78b564259de237e3e"}
Nov 03 16:02:15.235057 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.235043    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:15.236275 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.236253    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9" event={"ID":"646f7db2-f965-4cfc-883c-646eaf2975e5","Type":"ContainerStarted","Data":"0aae915ecc211a4be202cd7feddb0737d5142158deb334f9149b522b012792be"}
Nov 03 16:02:15.236366 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.236292    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9" event={"ID":"646f7db2-f965-4cfc-883c-646eaf2975e5","Type":"ContainerStarted","Data":"3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac"}
Nov 03 16:02:15.238110 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.238085    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q" event={"ID":"22b39777-54ae-4a95-91bb-81943254f1aa","Type":"ContainerStarted","Data":"da57b57b752ab0a8fb0c13ff13d697808fbcfe574fa60d48ac01771a82907a99"}
Nov 03 16:02:15.289168 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.289105    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kdkkw" podStartSLOduration=34.790731972 podStartE2EDuration="38.289088608s" totalImagesPullingTime="3.498356636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:15.288769569 +0000 UTC m=+38.960072179" watchObservedRunningTime="2026-11-03 16:02:15.289088608 +0000 UTC m=+38.960391217"
Nov 03 16:02:15.315636 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.315582    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q" podStartSLOduration=303.996473178 podStartE2EDuration="5m10.315565506s" totalImagesPullingTime="6.319092328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:15.314772029 +0000 UTC m=+38.986074655" watchObservedRunningTime="2026-11-03 16:02:15.315565506 +0000 UTC m=+38.986868114"
Nov 03 16:02:15.337443 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:15.337368    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9" podStartSLOduration=1.337353631 podStartE2EDuration="1.337353631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:15.335726343 +0000 UTC m=+39.007028976" watchObservedRunningTime="2026-11-03 16:02:15.337353631 +0000 UTC m=+39.008656239"
Nov 03 16:02:16.890509 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:16.890468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:16.891013 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:16.890520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:16.891013 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:16.890680    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:16.891013 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:16.890748    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.89073008 +0000 UTC m=+44.562032666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:16.891013 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:16.890681    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:02:16.891013 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:16.890836    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.890820019 +0000 UTC m=+44.562122629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : configmap "kube-rbac-proxy" not found
Nov 03 16:02:17.559873 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:17.559828    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mlccd_6a30fac5-ac40-4179-9628-fa3203ddb77b/dns-node-resolver/0.log"
Nov 03 16:02:17.568031 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:17.567992    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:17.568224 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:17.568097    2584 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:17.568546 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:17.568527    2584 scope.go:122] "RemoveContainer" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91"
Nov 03 16:02:17.568753 ip-10-0-133-129 kubenswrapper[2584]: E1103 16:02:17.568730    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4ftv2_openshift-console-operator(1b592ced-7c65-40a8-94d7-cad2c29fe081)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" podUID="1b592ced-7c65-40a8-94d7-cad2c29fe081"
Nov 03 16:02:18.253752 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:18.253719    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rtlt9" event={"ID":"d3387caf-55fb-4a86-9b89-f38c8fbe1915","Type":"ContainerStarted","Data":"6b6ad2a208712f80b16ba3b3d9a1a11b56d3729c2112e2360258c414a986c24e"}
Nov 03 16:02:18.272811 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:18.272753    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rtlt9" podStartSLOduration=33.510689224000004 podStartE2EDuration="40.272738086s" totalImagesPullingTime="6.762048862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:18.271935142 +0000 UTC m=+41.943237752" watchObservedRunningTime="2026-11-03 16:02:18.272738086 +0000 UTC m=+41.944040694"
Nov 03 16:02:18.359192 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:18.359116    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5wvsc_66a8e707-3e0b-42f2-988f-517d834a869c/node-ca/0.log"
Nov 03 16:02:18.419836 ip-10-0-133-129 kubenswrapper[2584]: I1103 16:02:18.419693    2584 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:02:18.419872 ip-10-0-133-129 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 16:02:18.440912 ip-10-0-133-129 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 16:02:18.441178 ip-10-0-133-129 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 16:02:18.441250 ip-10-0-133-129 systemd[1]: kubelet.service: Consumed 3.275s CPU time.
Nov 03 16:02:18.454280 ip-10-0-133-129 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 16:02:18.524274 ip-10-0-133-129 kubenswrapper[4964]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:02:18.524274 ip-10-0-133-129 kubenswrapper[4964]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 16:02:18.524274 ip-10-0-133-129 kubenswrapper[4964]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:02:18.524274 ip-10-0-133-129 kubenswrapper[4964]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:02:18.526519 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526425    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.526519 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526518    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.526519 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526522    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526525    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526528    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526531    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526533    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526536    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526538    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526541    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526547    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526555    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526559    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526562    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526565    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526568    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526571    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526573    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526576    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526579    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526582    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526585    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.526626 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526587    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526590    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526592    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526595    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526598    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526602    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526605    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526614    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526618    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526620    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526624    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526626    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526629    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526632    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526635    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526638    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526640    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526643    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526645    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.527134 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526648    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526651    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526653    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526656    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526659    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526662    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526665    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526667    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526671    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526673    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526676    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526678    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526682    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526684    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526687    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526690    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526693    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526695    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526698    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.527652 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526701    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526703    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526706    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526709    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526711    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526715    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526717    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526720    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526722    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526726    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526728    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526731    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526735    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526738    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526740    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526745    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526750    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526753    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526756    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526759    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.528117 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526762    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526765    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526767    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526770    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526773    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526775    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526778    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526780    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526783    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526786    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526788    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526791    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526794    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526796    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526799    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526802    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526804    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526807    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526809    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526812    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.528629 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526815    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526818    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526822    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526825    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526828    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.526831    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527637    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527645    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527648    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527651    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527654    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527659    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527662    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527665    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527667    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527670    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527673    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527675    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527679    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527681    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.529128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527685    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527688    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527690    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527693    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527696    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527699    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527701    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527704    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527706    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527709    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527712    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527714    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527718    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527721    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527724    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527727    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527731    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527735    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527739    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527741    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.529686 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527744    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527747    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527751    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527755    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527759    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527762    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527765    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527768    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527771    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527773    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527776    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527779    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527781    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527784    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527786    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527789    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527792    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527794    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527797    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.530184 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527800    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527802    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527805    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527808    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527811    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527813    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527816    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527819    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527821    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527824    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527826    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527829    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527832    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527834    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527837    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527839    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527842    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527844    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527847    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527850    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.530701 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527853    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527856    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527858    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527861    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527864    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527866    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527868    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527871    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527873    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527876    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527878    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527881    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527884    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527886    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527889    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527891    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527894    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527897    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527899    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.531183 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527902    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527905    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527907    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527910    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527914    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527916    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527919    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527921    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527924    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527926    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527928    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527931    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527934    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.527937    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528478    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528484    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528487    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528490    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528493    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.531689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528496    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528498    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528501    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528504    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528506    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528509    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528512    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528515    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528518    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528521    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528524    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528527    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528530    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528532    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528535    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528538    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528540    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528544    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528546    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528549    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.532186 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528552    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528554    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528556    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528560    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528562    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528567    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528572    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528575    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528577    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528580    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528583    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528585    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528589    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528591    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528594    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528597    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528599    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528601    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528604    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.532718 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528607    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528609    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528612    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528615    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528617    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528620    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528623    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528625    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528628    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528630    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528633    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528636    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528639    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528641    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528644    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528647    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528649    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528652    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528655    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528658    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.533170 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528660    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528663    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528666    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528669    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528671    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528675    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528677    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528680    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528683    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528685    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528688    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528691    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528693    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528696    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528698    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528701    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528703    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528706    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528709    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.533674 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528711    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528714    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528716    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528719    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528721    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528724    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528726    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528732    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528735    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528738    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528740    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528743    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528745    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528749    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528754    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528757    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528760    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528763    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528766    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.534128 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528768    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528771    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528773    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.528776    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528875    4964 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528890    4964 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528896    4964 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528901    4964 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528906    4964 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528910    4964 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528918    4964 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528923    4964 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528927    4964 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528930    4964 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528933    4964 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528937    4964 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528940    4964 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528943    4964 flags.go:64] FLAG: --cgroup-root=""
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528947    4964 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528950    4964 flags.go:64] FLAG: --client-ca-file=""
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528954    4964 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528957    4964 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528963    4964 flags.go:64] FLAG: --cluster-domain=""
Nov 03 16:02:18.534612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528966    4964 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528970    4964 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528973    4964 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528977    4964 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528982    4964 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528985    4964 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528988    4964 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528992    4964 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.528997    4964 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529000    4964 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529003    4964 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529006    4964 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529009    4964 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529014    4964 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529017    4964 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529020    4964 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529023    4964 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529026    4964 flags.go:64] FLAG: --enable-server="true"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529029    4964 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529034    4964 flags.go:64] FLAG: --event-burst="100"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529038    4964 flags.go:64] FLAG: --event-qps="50"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529041    4964 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529044    4964 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529047    4964 flags.go:64] FLAG: --eviction-hard=""
Nov 03 16:02:18.535155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529051    4964 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529055    4964 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529058    4964 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529061    4964 flags.go:64] FLAG: --eviction-soft=""
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529065    4964 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529069    4964 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529072    4964 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529075    4964 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529079    4964 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529082    4964 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529085    4964 flags.go:64] FLAG: --feature-gates=""
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529090    4964 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529093    4964 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529098    4964 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529101    4964 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529104    4964 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529109    4964 flags.go:64] FLAG: --help="false"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529112    4964 flags.go:64] FLAG: --hostname-override="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529115    4964 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529119    4964 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529122    4964 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529125    4964 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529129    4964 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 16:02:18.535795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529132    4964 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529135    4964 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529138    4964 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529142    4964 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529145    4964 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529148    4964 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529151    4964 flags.go:64] FLAG: --kube-reserved=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529154    4964 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529157    4964 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529160    4964 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529163    4964 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529167    4964 flags.go:64] FLAG: --lock-file=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529169    4964 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529172    4964 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529177    4964 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529187    4964 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529190    4964 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529193    4964 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529197    4964 flags.go:64] FLAG: --logging-format="text"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529200    4964 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529204    4964 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529207    4964 flags.go:64] FLAG: --manifest-url=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529210    4964 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529215    4964 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529219    4964 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 16:02:18.536367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529224    4964 flags.go:64] FLAG: --max-pods="110"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529228    4964 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529231    4964 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529234    4964 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529237    4964 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529240    4964 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529243    4964 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529246    4964 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529253    4964 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529257    4964 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529260    4964 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529263    4964 flags.go:64] FLAG: --pod-cidr=""
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529266    4964 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529269    4964 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529272    4964 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529275    4964 flags.go:64] FLAG: --port="10250"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529278    4964 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529281    4964 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-06e3420b04f089dfd"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529284    4964 flags.go:64] FLAG: --qos-reserved=""
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529287    4964 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529291    4964 flags.go:64] FLAG: --register-node="true"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529295    4964 flags.go:64] FLAG: --register-with-taints=""
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529299    4964 flags.go:64] FLAG: --registry-burst="10"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529302    4964 flags.go:64] FLAG: --registry-qps="5"
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529305    4964 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529308    4964 flags.go:64] FLAG: --reserved-memory=""
Nov 03 16:02:18.537033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529312    4964 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529315    4964 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529318    4964 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529322    4964 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529325    4964 flags.go:64] FLAG: --runonce="false"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529329    4964 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529334    4964 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529337    4964 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529340    4964 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529345    4964 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529349    4964 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529354    4964 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529359    4964 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529364    4964 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529368    4964 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529372    4964 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529375    4964 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529378    4964 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529381    4964 flags.go:64] FLAG: --system-cgroups=""
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529397    4964 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529404    4964 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529407    4964 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529410    4964 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529415    4964 flags.go:64] FLAG: --tls-min-version=""
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529418    4964 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 16:02:18.537823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529421    4964 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529425    4964 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529429    4964 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529432    4964 flags.go:64] FLAG: --v="2"
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529437    4964 flags.go:64] FLAG: --version="false"
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529441    4964 flags.go:64] FLAG: --vmodule=""
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529446    4964 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529449    4964 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529600    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529604    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529607    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529610    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529613    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529625    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529629    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529632    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529635    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529637    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529640    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529642    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529645    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529648    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.538465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529650    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529653    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529656    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529658    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529661    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529663    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529666    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529669    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529671    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529674    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529677    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529679    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529683    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529686    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529689    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529692    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529695    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529698    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529701    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.539333 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529703    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529706    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529709    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529711    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529715    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529723    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529727    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529730    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529733    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529735    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529738    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529740    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529743    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529746    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529748    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529751    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529754    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529756    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529759    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529761    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.540143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529764    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529766    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529769    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529772    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529775    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529779    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529781    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529785    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529787    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529791    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529793    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529796    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529800    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529803    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529806    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529809    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529814    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529818    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529822    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.540814 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529825    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529827    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529830    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529833    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529835    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529838    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529840    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529843    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529846    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529848    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529851    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529853    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529856    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529859    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529861    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529864    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529866    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529869    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529873    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.541409 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529876    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529878    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529881    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529884    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529886    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529889    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529891    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529894    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529897    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529899    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529903    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529907    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529909    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529912    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529915    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.529941    4964 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:02:18.542086 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529983    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529988    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529991    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529994    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.529997    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530000    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530002    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530005    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530008    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530011    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530013    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530016    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530019    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530021    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530024    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530028    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530031    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530034    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530036    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530039    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.542540 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530042    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530045    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530047    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530050    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530052    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530055    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530057    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530061    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530064    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530066    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530070    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530072    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530075    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530078    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530080    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530083    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530085    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530088    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530090    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.543026 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530093    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530096    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530098    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530101    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530103    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530106    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530108    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530111    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530115    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530117    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530120    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530123    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530125    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530128    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530130    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530133    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530135    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530138    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530140    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530143    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.543634 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530148    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530151    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530153    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530156    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530159    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530162    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530164    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530167    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530170    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530172    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530175    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530178    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530180    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530183    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530186    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530189    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530193    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530195    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530198    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.544412 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530201    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530209    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530213    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530216    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530219    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530223    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530227    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530230    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530233    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530235    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530238    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530240    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530243    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530247    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530250    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530253    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530255    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530258    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530261    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.544901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530264    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530267    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530270    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530272    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530275    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530277    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530281    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530284    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.530286    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.534291    4964 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.534309    4964 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.534328    4964 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.534333    4964 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534418    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534424    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534428    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534431    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534434    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534437    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534440    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.545369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534443    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534445    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534448    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534451    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534454    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534457    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534459    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534462    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534465    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534468    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534471    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534476    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534479    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534481    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534484    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534486    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534489    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534492    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534495    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534497    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.545873 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534500    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534503    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534506    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534509    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534512    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534514    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534518    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534520    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534523    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534526    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534529    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534532    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534534    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534537    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534540    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534542    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534545    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534548    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534550    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534552    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.546366 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534555    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534558    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534561    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534563    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534567    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534570    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534573    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534576    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534578    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534581    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534584    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534587    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534589    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534592    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534594    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534597    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534599    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534603    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534607    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.546899 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534610    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534613    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534616    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534619    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534623    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534627    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534631    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534634    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534637    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534640    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534643    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534645    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534648    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534651    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534653    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534656    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534659    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534661    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534665    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.547508 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534668    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534671    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534673    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534676    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534678    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534681    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534684    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534687    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534689    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534692    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534695    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534697    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534700    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534703    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534705    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534708    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534711    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534713    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534716    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.548148 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534718    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534721    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.534749    4964 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534901    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534906    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534909    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534912    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534915    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534917    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534920    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534924    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534927    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534930    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534933    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534936    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534939    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.548869 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534941    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534944    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534946    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534949    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534951    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534954    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534957    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534959    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534962    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534965    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534968    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534970    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534973    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534976    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534978    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534981    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534983    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534986    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534988    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534991    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.549421 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534994    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534996    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.534999    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535001    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535004    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535006    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535009    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535012    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535014    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535017    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535020    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535023    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535026    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535029    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535031    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535034    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535036    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535039    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535042    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535044    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.549951 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535047    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535050    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535052    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535055    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535058    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535060    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535063    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535066    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535068    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535071    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535074    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535076    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535079    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535081    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535084    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535086    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535089    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535091    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535094    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.550465 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535097    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535099    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535102    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535105    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535109    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535111    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535114    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535117    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535119    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535122    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535124    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535127    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535129    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535132    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535136    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535139    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535143    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535145    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535149    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.551009 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535151    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535154    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535157    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535160    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535162    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535165    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535168    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535171    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535174    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535176    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535179    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535182    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535184    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535187    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535190    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.535218    4964 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:02:18.551502 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535269    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535274    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535279    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535283    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535287    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535291    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535296    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535300    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535304    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535308    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535312    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535316    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535320    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535324    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535328    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535332    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535337    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535340    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535345    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535349    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:02:18.551900 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535352    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535356    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535364    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535368    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535374    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535380    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535399    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535404    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535409    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535413    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535417    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535421    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535425    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535429    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535433    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535438    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535442    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535447    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535450    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:02:18.552456 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535456    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535460    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535464    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535475    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535480    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535484    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535488    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535492    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535496    4964 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535500    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535504    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535508    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535514    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535520    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535524    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535531    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535535    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535540    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535544    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:02:18.552931 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535548    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535552    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535555    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535558    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535560    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535563    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535566    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535569    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535572    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535574    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535578    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535581    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535584    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535587    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535589    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535592    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535595    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535599    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535602    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:02:18.553521 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535605    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535608    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535610    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535613    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535615    4964 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535618    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535621    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535623    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535626    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535630    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535633    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535637    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535640    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535642    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535645    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535648    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535650    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535653    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535655    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535658    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:02:18.553984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535660    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535663    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535665    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535668    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535671    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535674    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535676    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535679    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:18.535681    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.535754    4964 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.536774    4964 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.536849    4964 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.537043    4964 server.go:1014] "Starting client certificate rotation"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.537194    4964 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.537260    4964 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:56:37 +0000 UTC" deadline="2028-08-20 01:39:07.419214496 +0000 UTC"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.537299    4964 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15729h36m48.88191864s"
Nov 03 16:02:18.554500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.537505    4964 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:02:18.555147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.539241    4964 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:02:18.555147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.541410    4964 log.go:25] "Validated CRI v1 runtime API"
Nov 03 16:02:18.555147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.543150    4964 log.go:25] "Validated CRI v1 image API"
Nov 03 16:02:18.555147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.543872    4964 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 16:02:18.555147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.545967    4964 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8170ca1b-4cfa-4313-bc63-5435016a1192:/dev/nvme0n1p4 a3b99ce7-fd0c-47d3-ba86-fbd6b4a8eedc:/dev/nvme0n1p3]
Nov 03 16:02:18.555611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.545990    4964 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9f3a0675540a229942cbffce98a4240752b7560cf2fba953a01a8095eaf655a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9f3a0675540a229942cbffce98a4240752b7560cf2fba953a01a8095eaf655a/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a088fef-29ab-4b9e-a90d-f6560aa4ee43/volumes/kubernetes.io~projected/kube-api-access-7pqfk:{mountpoint:/var/lib/kubelet/pods/0a088fef-29ab-4b9e-a90d-f6560aa4ee43/volumes/kubernetes.io~projected/kube-api-access-7pqfk major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~projected/kube-api-access-5ftxg:{mountpoint:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~projected/kube-api-access-5ftxg major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b592ced-7c65-40a8-94d7-cad2c29fe081/volumes/kubernetes.io~projected/kube-api-access-cr5lp:{mountpoint:/var/lib/kubelet/pods/1b592ced-7c65-40a8-94d7-cad2c29fe081/volumes/kubernetes.io~projected/kube-api-access-cr5lp major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b592ced-7c65-40a8-94d7-cad2c29fe081/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1b592ced-7c65-40a8-94d7-cad2c29fe081/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c0598b6-8a65-4e35-9347-907aa42e8781/volumes/kubernetes.io~projected/kube-api-access-2l9ch:{mountpoint:/var/lib/kubelet/pods/1c0598b6-8a65-4e35-9347-907aa42e8781/volumes/kubernetes.io~projected/kube-api-access-2l9ch major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22b39777-54ae-4a95-91bb-81943254f1aa/volumes/kubernetes.io~projected/kube-api-access-rr7g4:{mountpoint:/var/lib/kubelet/pods/22b39777-54ae-4a95-91bb-81943254f1aa/volumes/kubernetes.io~projected/kube-api-access-rr7g4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~projected/kube-api-access-6x7tb:{mountpoint:/var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~projected/kube-api-access-6x7tb major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~secret/default-certificate major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/315b5fc8-0084-4395-96a5-8ae4e210b974/volumes/kubernetes.io~projected/kube-api-access-vkzkr:{mountpoint:/var/lib/kubelet/pods/315b5fc8-0084-4395-96a5-8ae4e210b974/volumes/kubernetes.io~projected/kube-api-access-vkzkr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4beaa779-098d-4007-88f8-25796f1758f0/volumes/kubernetes.io~projected/kube-api-access-kw5tr:{mountpoint:/var/lib/kubelet/pods/4beaa779-098d-4007-88f8-25796f1758f0/volumes/kubernetes.io~projected/kube-api-access-kw5tr major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/646f7db2-f965-4cfc-883c-646eaf2975e5/volumes/kubernetes.io~projected/kube-api-access-2pxw4:{mountpoint:/var/lib/kubelet/pods/646f7db2-f965-4cfc-883c-646eaf2975e5/volumes/kubernetes.io~projected/kube-api-access-2pxw4 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/646f7db2-f965-4cfc-883c-646eaf2975e5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/646f7db2-f965-4cfc-883c-646eaf2975e5/volumes/kubernetes.io~secret/signing-key major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7/volumes/kubernetes.io~projected/kube-api-access-kl27k:{mountpoint:/var/lib/kubelet/pods/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7/volumes/kubernetes.io~projected/kube-api-access-kl27k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66a8e707-3e0b-42f2-988f-517d834a869c/volumes/kubernetes.io~projected/kube-api-access-drxvl:{mountpoint:/var/lib/kubelet/pods/66a8e707-3e0b-42f2-988f-517d834a869c/volumes/kubernetes.io~projected/kube-api-access-drxvl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/volumes/kubernetes.io~projected/kube-api-access-b2v2p:{mountpoint:/var/lib/kubelet/pods/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/volumes/kubernetes.io~projected/kube-api-access-b2v2p major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a30fac5-ac40-4179-9628-fa3203ddb77b/volumes/kubernetes.io~projected/kube-api-access-vw4vs:{mountpoint:/var/lib/kubelet/pods/6a30fac5-ac40-4179-9628-fa3203ddb77b/volumes/kubernetes.io~projected/kube-api-access-vw4vs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2e92347-295e-4c8e-bef8-519fa58d4e87/volumes/kubernetes.io~projected/kube-api-access-n8qbn:{mountpoint:/var/lib/kubelet/pods/a2e92347-295e-4c8e-bef8-519fa58d4e87/volumes/kubernetes.io~projected/kube-api-access-n8qbn major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2e92347-295e-4c8e-bef8-519fa58d4e87/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a2e92347-295e-4c8e-bef8-519fa58d4e87/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a467d9db-7d44-47e3-a142-30cb089a882b/volumes/kubernetes.io~projected/kube-api-access-k9v5s:{mountpoint:/var/lib/kubelet/pods/a467d9db-7d44-47e3-a142-30cb089a882b/volumes/kubernetes.io~projected/kube-api-access-k9v5s major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a71be33d-a6e5-4b3a-b703-64cc2f214f5b/volumes/kubernetes.io~projected/kube-api-access-rw48k:{mountpoint:/var/lib/kubelet/pods/a71be33d-a6e5-4b3a-b703-64cc2f214f5b/volumes/kubernetes.io~projected/kube-api-access-rw48k major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a71be33d-a6e5-4b3a-b703-64cc2f214f5b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a71be33d-a6e5-4b3a-b703-64cc2f214f5b/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198/volumes/kubernetes.io~projected/kube-api-access-gmrnn:{mountpoint:/var/lib/kubelet/pods/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198/volumes/kubernetes.io~projected/kube-api-access-gmrnn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volumes/kubernetes.io~projected/kube-api-access-dklmt:{mountpoint:/var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volumes/kubernetes.io~projected/kube-api-access-dklmt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~projected/kube-api-access-djljv:{mountpoint:/var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~projected/kube-api-access-djljv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/volumes/kubernetes.io~projected/kube-api-access-6vvlj:{mountpoint:/var/lib/kubelet/pods/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/volumes/kubernetes.io~projected/kube-api-access-6vvlj major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d327c29c-36d3-4d01-8cab-bf10a76dbf48/volumes/kubernetes.io~projected/kube-api-access-kvhp2:{mountpoint:/var/lib/kubelet/pods/d327c29c-36d3-4d01-8cab-bf10a76dbf48/volumes/kubernetes.io~projected/kube-api-access-kvhp2 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3387caf-55fb-4a86-9b89-f38c8fbe1915/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d3387caf-55fb-4a86-9b89-f38c8fbe1915/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a6ba52-20d8-40fe-9723-54418819a3ab/volumes/kubernetes.io~projected/kube-api-access-w5dp9:{mountpoint:/var/lib/kubelet/pods/d7a6ba52-20d8-40fe-9723-54418819a3ab/volumes/kubernetes.io~projected/kube-api-access-w5dp9 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da622b12-25ac-43dc-8fbb-c9a974aa9f9d/volumes/kubernetes.io~projected/kube-api-access-jq4zf:{mountpoint:/var/lib/kubelet/pods/da622b12-25ac-43dc-8fbb-c9a974aa9f9d/volumes/kubernetes.io~projected/kube-api-access-jq4zf major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~projected/kube-api-access-2z9tc:{mountpoint:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~projected/kube-api-access-2z9tc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfe05141-01c7-4446-a5b5-e61abe65dfa2/volumes/kubernetes.io~projected/kube-api-access-w8jl7:{mountpoint:/var/lib/kubelet/pods/dfe05141-01c7-4446-a5b5-e61abe65dfa2/volumes/kubernetes.io~projected/kube-api-access-w8jl7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3c566eb-b54b-47e7-a848-c1cacacc63e7/volumes/kubernetes.io~projected/kube-api-access-8262d:{mountpoint:/var/lib/kubelet/pods/e3c566eb-b54b-47e7-a848-c1cacacc63e7/volumes/kubernetes.io~projected/kube-api-access-8262d major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3dbcf1c-4684-46e8-b806-1df38410a5c9/volumes/kubernetes.io~projected/kube-api-access-nj857:{mountpoint:/var/lib/kubelet/pods/e3dbcf1c-4684-46e8-b806-1df38410a5c9/volumes/kubernetes.io~projected/kube-api-access-nj857 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3dbcf1c-4684-46e8-b806-1df38410a5c9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e3dbcf1c-4684-46e8-b806-1df38410a5c9/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3bb7151-48a1-4b4d-9784-31da848ac4f0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f3bb7151-48a1-4b4d-9784-31da848ac4f0/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2/volumes/kubernetes.io~projected/kube-api-access-kngjp:{mountpoint:/var/lib/kubelet/pods/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2/volumes/kubernetes.io~projected/kube-api-access-kngjp major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe4abe85-5418-426a-bb01-57b63c1f5c52/volumes/kubernetes.io~projected/kube-api-access-g74rn:{mountpoint:/var/lib/kubelet/pods/fe4abe85-5418-426a-bb01-57b63c1f5c52/volumes/kubernetes.io~projected/kube-api-access-g74rn major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d03c80da7a203be3a6e5e0b4b5e305345eaa76c17ac98adda26ffc8c3716a5b5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f4ac3608016048cfcc62896eb8f6db2e17922d773c669c5cf3a8893d99a340f0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0117fbf17fc9bffe33b494d83495aa6c5e8414a0d99ec71669a84157fa000a8c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/4ad5681e37d960d75b502b3755d3a0bb9b37417be0cb3e9ac1b0077091646213/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/722aaa129279e29c283fd39d19b74b284fee74549a451201b2671cdab5cd9887/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/974d16fcba82595e7a2b200467a2439e812d552cc0521e74eaf1d6d2094d7726/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bf533084e72973ab7c56394fa2ec100410422f7ef2b45e648d55bd062cba325d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f506a4c37ecb417ded4b65b71ded34c79c60b31f17fe5ee3f913700b5bdc46eb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/87e57dd735a2f323185f897efa50de0579aba018d77e66bd4183173f82c5084d/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b272d867cd8b9bb0b1a09bf341e4d73c016a480ab963076345e714b1a5699135/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/224f198c76ccb058242f6cbe4e50500d5f24c59c82e55b19605f395d36c00517/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/919b5b26df8c89fa92c6a39360d7a88b7e4748d04881e293a2b7f4db6c3aa7f5/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c673e43416f24f2d250093b76afeef174d2e1278199409eda5ea01cc655d48d2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/e3099a3c15e39e85c7320c4c86ae407b73d5bdd51e0551b055990e54a6322531/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3c4b651587b104bcdf72f6c086a9ea1292bfa5ddb9dfa1aea9218a1b3b04f577/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2471981654ff612d901136cd17b9e00bcf3a3aec06dbba70f1820bc8a71828ca/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2187ec4db71b273eb741b817388148cbf5363805f5296e69293a32ccbc13a3bd/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/43f81d78a5baf4400986f88051d945f52d4168a4e2e3ef2866e09ce78ad6d485/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/579ffd84a529a71f421eb8b5f4d96fccabceb0c80e4c32179d681ec73241e2dc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/336c209fafeb0148e4d0a4522ba9e048aeed1ecececb3dec6072ab0886d57492/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4f7b4565cafa08d1fb98d02a11eb99c03e487fb6a92a4fac7c06094b4cbc9803/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/e5ed6cebf4e9c6e124963bc968e1f74198af229e90c4f4d2407c5615243a440b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b3a513a212009fac359677b176ca983ac608649ccad241b427cf9cbeeb929b51/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/4e1ccb4bddbfef21180405074d08f117d05edaf0a37fd9c08682bb7449a0096b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/bcdafb12fb9bb4faca4659e154c22b362388100902ab9df5d39ef4225b327001/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/af318796eba91da988afa1a283b69a0c23515813e12943c08c74a52437a1a2f9/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6c6b4f09792f77aa2994dbf29ba1beeef476a5c31b2b2d378004e080a6495b32/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/458accf0a9acd17cd9045ac8aeb7b926db8a1187de03a3f46faba4e4aa6bc02d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/c74f1bdaebad228eeb8244ef855a97292dde3405019103e8a9cbcc58674975ee/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/71a331d9078e427c17eab87ade581a636bf47f641d2937a160be873c96eb2af1/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e1bd39a956995304990dda67de5b57a6fd15ddfa0919da619add3da60120dc81/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/6b0d737ae069ebccc1ff71c347cd2f217879f611a89db3695d366aea32683bc4/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/ff2f2c8df5c427e79fd6670a3332230613fe35c0d99e92f4d43525c79b2aab4b/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/99697f75d973f88a15c26036074707ca421710be74a81ccfcf7cce7ad0fbe2a9/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/8120e3c96757ec2547459e8c2ea28979368791c65385f8b38112774cfa938654/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1521fd0471256e309b31fcf7ca332ed52f2d03a602d9112bbe442bd5b78b5c7f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/9660bedb7005b6fa9625560ffe5c26964113646bd75480d7f9ae8a6efdc30a12/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0fc85214055eb997c78a433f9c0b2f151eff35b50335d4df542bde3415f14637/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/dde15d3e80c47185c3cee1376f593f3e487acea6c13a3a5a355bd7c2429ee0a1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/fdfd340c8202df1d55c78ce5ec6d41242a5d6eb77ec1196f30de218dcdf3b8c9/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/487f416f4b0994c931802a959c13881f106978b492a8bec42641f5a8ec6dc470/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/2c7e2eb72c56725ea3f3f282498243c924f8faae1f134e5854169df55abd2e07/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/c4dac15c3f5c9da20dc084415124eaefa42a4e7713a6e7e0fd42b22b8dc83365/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/a37820e4176bbb11f2990e8c8487da211da0ac65da2eda5f4ad5d40b692263e6/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/a9276774973296834a593a10f319f0c8d19fd7df2c63e514248860b4ce31e82b/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/ca3cc0283e7ed5ba0f38269480ba7859b0cef0df563c6b503eeb3ce2211d2ce1/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a71446a967305b792f92d7e4ce8fa5bd51e714926dd7788831cc1bc471ee2049/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/fbda7ac199a32fabe27b76b7c5247e00e8bb06b6de7af2e343d36c9095f2834d/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9b261605249ef15f7b5917c508d360f22aa18da4f4a193b7ff5eddd41914ddaa/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/846d7548c514fd9d0a0cd1192cdf1bccae23d93bd923fcd3c2b418db21795de2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c7435a4ebbf5b33461e0ace0fa300fdc876abfcd392215f506cb759c1460d4d2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/faaffee32a4c023ef067f754d195cf9255c88d2594c3f38da379acac9551c516/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/f463ef4ef09a9c1c2ec1ac23e91ee9e83f62c1e44385d584d3afd870fe5c14ec/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e387b2ab54781fa7e622b033eba3c52a6376fd330e2bc25881bc491fc8da33ef/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/47a9632db59194e16c2404529706a71f31fe29bcb2f19fa74098ac5d36a026e1/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9b4ebc996fcace6d321dd772270da10215773838fc4dbdeb5df8f1907d666cd6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/c311320536210b9fb9f642e6a61b881a8eea788a0d386ccd04b9d07d4bcb68da/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b842e1ef621731a1e6788671019663318a2504f6e743c69bb11940b6796c32ee/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.554063    4964 manager.go:217] Machine: {Timestamp:2026-11-03 16:02:18.553585536 +0000 UTC m=+0.066967308 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101394 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22dde31f7a1df68c4fd6dd8956b349 SystemUUID:ec22dde3-1f7a-1df6-8c4f-d6dd8956b349 BootID:efdbc630-18ae-43bb-b561-63a5d3cd46b4 Filesystems:[{Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da622b12-25ac-43dc-8fbb-c9a974aa9f9d/volumes/kubernetes.io~projected/kube-api-access-jq4zf DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7/volumes/kubernetes.io~projected/kube-api-access-kl27k DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dfe05141-01c7-4446-a5b5-e61abe65dfa2/volumes/kubernetes.io~projected/kube-api-access-w8jl7 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3bb7151-48a1-4b4d-9784-31da848ac4f0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a6ba52-20d8-40fe-9723-54418819a3ab/volumes/kubernetes.io~projected/kube-api-access-w5dp9 DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/751af3e065bda57702458fbced09f3436cf98bfb88cf500d51596d2b39b70df6/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3387caf-55fb-4a86-9b89-f38c8fbe1915/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c0598b6-8a65-4e35-9347-907aa42e8781/volumes/kubernetes.io~projected/kube-api-access-2l9ch DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/volumes/kubernetes.io~projected/kube-api-access-6vvlj DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b592ced-7c65-40a8-94d7-cad2c29fe081/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe4abe85-5418-426a-bb01-57b63c1f5c52/volumes/kubernetes.io~projected/kube-api-access-g74rn DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bfa4135b3fd420483aaf5616d38c3917ba152ba8df251305fc4a469b243d5d4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3dbcf1c-4684-46e8-b806-1df38410a5c9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2e92347-295e-4c8e-bef8-519fa58d4e87/volumes/kubernetes.io~projected/kube-api-access-n8qbn DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~projected/kube-api-access-2z9tc DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e81d5c6d25c05060adc4c4da24c00becd1f6d0b53558e6bb99ecf5dbc1843e54/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a467d9db-7d44-47e3-a142-30cb089a882b/volumes/kubernetes.io~projected/kube-api-access-k9v5s DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/646f7db2-f965-4cfc-883c-646eaf2975e5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c5e1c18ecb47f3c9331ff00fa0bc728eb279a0c156d151ecd9a61159a5a8aac/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66a8e707-3e0b-42f2-988f-517d834a869c/volumes/kubernetes.io~projected/kube-api-access-drxvl DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce398f3164527e413d8906bf497bbcf2a5f39f349352e99d88fbcb8bf43305b8/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b157a6035a5c975ae9e9f76838d24674b41230cd08b3550ed5b07efac9b9ddb/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2e92347-295e-4c8e-bef8-519fa58d4e87/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a274b1a93bd7ab62e714b5267fce75a4c3bac73a8c10a8323fdb4122e9f23376/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volumes/kubernetes.io~projected/kube-api-access-dklmt DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9f3a0675540a229942cbffce98a4240752b7560cf2fba953a01a8095eaf655a/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b592ced-7c65-40a8-94d7-cad2c29fe081/volumes/kubernetes.io~projected/kube-api-access-cr5lp DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d96953a77a8481bb17597614a018e55825befed8c3325fc93135eaf0d148b356/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d73c404bb066853813232cdd65dbc3d0bd27c50cd31be3a1a3a219d035901915/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a088fef-29ab-4b9e-a90d-f6560aa4ee43/volumes/kubernetes.io~projected/kube-api-access-7pqfk DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a30fac5-ac40-4179-9628-fa3203ddb77b/volumes/kubernetes.io~projected/kube-api-access-vw4vs DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3dbcf1c-4684-46e8-b806-1df38410a5c9/volumes/kubernetes.io~projected/kube-api-access-nj857 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198/volumes/kubernetes.io~projected/kube-api-access-gmrnn DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/557b309e4504ac9977667af3faffaaac53002aae7265dad918cc610c230d1625/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4beaa779-098d-4007-88f8-25796f1758f0/volumes/kubernetes.io~projected/kube-api-access-kw5tr DeviceMajor:0 DeviceMinor:311 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/315b5fc8-0084-4395-96a5-8ae4e210b974/volumes/kubernetes.io~projected/kube-api-access-vkzkr DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb2ef07fc2857aecea7b7c99fe53e72700beb3d07831112d60282fed3a57cb82/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4965fc63a97774d01ae0420b2edea1ec7b6bf50b363e0d2b4f2f72831daf7973/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a71be33d-a6e5-4b3a-b703-64cc2f214f5b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22b39777-54ae-4a95-91bb-81943254f1aa/volumes/kubernetes.io~projected/kube-api-access-rr7g4 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/volumes/kubernetes.io~projected/kube-api-access-b2v2p DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb8ba13e261f5e8c600626443ea8389fee07b4803a9797f15feba413c975a4f6/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3396dc722a290d872d2bd4b89b7387ce08cbcafa2a9e64b475614c213ab8765c/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb07a583d41832308a93363b8e23ee912f75a6421caec5835eba0d570e08cdd1/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/646f7db2-f965-4cfc-883c-646eaf2975e5/volumes/kubernetes.io~projected/kube-api-access-2pxw4 DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes/kubernetes.io~projected/kube-api-access-5ftxg DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d327c29c-36d3-4d01-8cab-bf10a76dbf48/volumes/kubernetes.io~projected/kube-api-access-kvhp2 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2351ca8a-ffba-4733-8413-2ae0b585b489/volumes/kubernetes.io~projected/kube-api-access-6x7tb DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f31d0bd84191951fa485d73fcc66241db80edb76fdf9fd4587a7c0942801513/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed5336085285248beda6ee7ac64289759de3634c89316c104b41204ce6530860/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2/volumes/kubernetes.io~projected/kube-api-access-kngjp DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a71be33d-a6e5-4b3a-b703-64cc2f214f5b/volumes/kubernetes.io~projected/kube-api-access-rw48k DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3c566eb-b54b-47e7-a848-c1cacacc63e7/volumes/kubernetes.io~projected/kube-api-access-8262d DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b903185c-6b3b-45dd-aac2-caaa87d0b075/volumes/kubernetes.io~projected/kube-api-access-djljv DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/085a8876827fe3ecfb30c81b95cf458666d6e102f5aca20d7f7a4bc6b955e3f8/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3538cb6200f86c3f02178019ace20e5b4283dc22ccb7f3e1b2bdb3ada8c48d52/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d99334d9190fe005faa1a096bed7f1bb2b06fbed6962777fd56b5a94da78657e/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3538cb6200f86c3 MacAddress:46:1e:2a:dc:63:20 Speed:10000 Mtu:8901} {Name:3c5e1c18ecb47f3 MacAddress:ca:63:69:c2:c5:21 Speed:10000 Mtu:8901} {Name:3f31d0bd8419195 MacAddress:3a:8d:75:83:df:2b Speed:10000 Mtu:8901} {Name:6b157a6035a5c97 MacAddress:f6:1d:43:07:2f:8e Speed:10000 Mtu:8901} {Name:94f497c6f15eab0 MacAddress:66:81:c9:c5:2a:bb Speed:10000 Mtu:8901} {Name:a274b1a93bd7ab6 MacAddress:5e:6e:1e:3d:6d:36 Speed:10000 Mtu:8901} {Name:bb2ef07fc2857ae MacAddress:d2:56:15:d3:84:12 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f0:5f:ee:85 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:78:4a:39:df:44 Speed:0 Mtu:8901} {Name:cb8ba13e261f5e8 MacAddress:b2:ef:21:66:6a:72 Speed:10000 Mtu:8901} {Name:ce398f3164527e4 MacAddress:02:15:9b:0b:5d:8a Speed:10000 Mtu:8901} {Name:d73c404bb066853 MacAddress:22:44:86:83:ec:57 Speed:10000 Mtu:8901} {Name:d96953a77a8481b MacAddress:72:9e:05:26:5c:d3 Speed:10000 Mtu:8901} {Name:d99334d9190fe00 MacAddress:2a:a4:59:1c:59:fa Speed:10000 Mtu:8901} {Name:e81d5c6d25c0506 MacAddress:52:a4:06:9f:9b:a3 Speed:10000 Mtu:8901} {Name:eb07a583d418323 MacAddress:52:22:06:a2:e4:92 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f0:5f:ee:85 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:15:f4:50:f4:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.554471    4964 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.554528    4964 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.554930    4964 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.554961    4964 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-129.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555176    4964 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555189    4964 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555208    4964 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555227    4964 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555257    4964 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555536    4964 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555718    4964 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555764    4964 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555778    4964 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555792    4964 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555806    4964 kubelet.go:405] "Adding apiserver pod source"
Nov 03 16:02:18.555993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.555820    4964 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 16:02:18.557413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.557358    4964 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 16:02:18.557413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.557398    4964 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:02:18.558290 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.558219    4964 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 16:02:18.561677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.561660    4964 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 16:02:18.561677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.561678    4964 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:02:18.561790 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.561780    4964 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 16:02:18.562000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.561988    4964 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 16:02:18.562030 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562018    4964 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 16:02:18.562148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562138    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 16:02:18.562187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562179    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562189    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562195    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562201    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562207    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562213    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562223    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 16:02:18.562228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562230    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 16:02:18.562454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562237    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 16:02:18.562454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562246    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 16:02:18.562454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562256    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 16:02:18.562454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562282    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 16:02:18.562454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.562289    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 16:02:18.565961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.565947    4964 server.go:1263] "Started kubelet"
Nov 03 16:02:18.566074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.566042    4964 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 16:02:18.566138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.566052    4964 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 16:02:18.566138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.566132    4964 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 16:02:18.566567 ip-10-0-133-129 systemd[1]: Started Kubernetes Kubelet.
Nov 03 16:02:18.566729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.566648    4964 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 16:02:18.567369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.567341    4964 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 16:02:18.569640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.569623    4964 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 16:02:18.570255 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.570228    4964 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 16:02:18.570723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.570708    4964 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 16:02:18.570822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.570812    4964 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 16:02:18.571207 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.571193    4964 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 16:02:18.571655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.571615    4964 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:56:39 +0000 UTC" deadline="2028-06-07 15:03:12.602123769 +0000 UTC"
Nov 03 16:02:18.571655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.571654    4964 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13967h0m54.030472976s"
Nov 03 16:02:18.572522 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.572484    4964 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-129.us-east-2.compute.internal\" not found"
Nov 03 16:02:18.575778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575587    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a467d9db-7d44-47e3-a142-30cb089a882b" volumeName="kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-multus-daemon-config" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575787    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5dea52a-10d2-4ca8-a4e9-096f66ad73fc" volumeName="kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-env-overrides" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575811    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575829    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a30fac5-ac40-4179-9628-fa3203ddb77b" volumeName="kubernetes.io/empty-dir/6a30fac5-ac40-4179-9628-fa3203ddb77b-tmp-dir" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575843    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a30fac5-ac40-4179-9628-fa3203ddb77b" volumeName="kubernetes.io/projected/6a30fac5-ac40-4179-9628-fa3203ddb77b-kube-api-access-vw4vs" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575861    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5dea52a-10d2-4ca8-a4e9-096f66ad73fc" volumeName="kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575875    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b903185c-6b3b-45dd-aac2-caaa87d0b075" volumeName="kubernetes.io/projected/b903185c-6b3b-45dd-aac2-caaa87d0b075-kube-api-access-djljv" seLinuxMountContext=""
Nov 03 16:02:18.575909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575903    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" volumeName="kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575932    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="646f7db2-f965-4cfc-883c-646eaf2975e5" volumeName="kubernetes.io/configmap/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-cabundle" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575958    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a467d9db-7d44-47e3-a142-30cb089a882b" volumeName="kubernetes.io/projected/a467d9db-7d44-47e3-a142-30cb089a882b-kube-api-access-k9v5s" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575973    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.575987    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3dbcf1c-4684-46e8-b806-1df38410a5c9" volumeName="kubernetes.io/secret/e3dbcf1c-4684-46e8-b806-1df38410a5c9-cert" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576006    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe4abe85-5418-426a-bb01-57b63c1f5c52" volumeName="kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576021    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="646f7db2-f965-4cfc-883c-646eaf2975e5" volumeName="kubernetes.io/projected/646f7db2-f965-4cfc-883c-646eaf2975e5-kube-api-access-2pxw4" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576047    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576063    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576083    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b592ced-7c65-40a8-94d7-cad2c29fe081" volumeName="kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-config" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576098    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a71be33d-a6e5-4b3a-b703-64cc2f214f5b" volumeName="kubernetes.io/secret/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-serving-cert" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576112    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5dea52a-10d2-4ca8-a4e9-096f66ad73fc" volumeName="kubernetes.io/projected/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-kube-api-access-dklmt" seLinuxMountContext=""
Nov 03 16:02:18.576165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576130    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" volumeName="kubernetes.io/projected/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-kube-api-access-6vvlj" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576143    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3387caf-55fb-4a86-9b89-f38c8fbe1915" volumeName="kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576239    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576269    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a6ba52-20d8-40fe-9723-54418819a3ab" volumeName="kubernetes.io/projected/d7a6ba52-20d8-40fe-9723-54418819a3ab-kube-api-access-w5dp9" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576293    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfe05141-01c7-4446-a5b5-e61abe65dfa2" volumeName="kubernetes.io/projected/dfe05141-01c7-4446-a5b5-e61abe65dfa2-kube-api-access-w8jl7" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576309    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3bb7151-48a1-4b4d-9784-31da848ac4f0" volumeName="kubernetes.io/secret/f3bb7151-48a1-4b4d-9784-31da848ac4f0-agent-certs" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576337    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2" volumeName="kubernetes.io/projected/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2-kube-api-access-kngjp" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576352    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe4abe85-5418-426a-bb01-57b63c1f5c52" volumeName="kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576370    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe4abe85-5418-426a-bb01-57b63c1f5c52" volumeName="kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-binary-copy" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576425    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe4abe85-5418-426a-bb01-57b63c1f5c52" volumeName="kubernetes.io/projected/fe4abe85-5418-426a-bb01-57b63c1f5c52-kube-api-access-g74rn" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576446    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e92347-295e-4c8e-bef8-519fa58d4e87" volumeName="kubernetes.io/secret/a2e92347-295e-4c8e-bef8-519fa58d4e87-serving-cert" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576461    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5dea52a-10d2-4ca8-a4e9-096f66ad73fc" volumeName="kubernetes.io/secret/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576474    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" volumeName="kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-snapshots" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576487    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576498    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b592ced-7c65-40a8-94d7-cad2c29fe081" volumeName="kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-trusted-ca" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576509    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5dea52a-10d2-4ca8-a4e9-096f66ad73fc" volumeName="kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-config" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576518    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b903185c-6b3b-45dd-aac2-caaa87d0b075" volumeName="kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-tmp" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576528    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da622b12-25ac-43dc-8fbb-c9a974aa9f9d" volumeName="kubernetes.io/projected/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-kube-api-access-jq4zf" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576539    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576547    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3c566eb-b54b-47e7-a848-c1cacacc63e7" volumeName="kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576558    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576567    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b592ced-7c65-40a8-94d7-cad2c29fe081" volumeName="kubernetes.io/projected/1b592ced-7c65-40a8-94d7-cad2c29fe081-kube-api-access-cr5lp" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576576    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a467d9db-7d44-47e3-a142-30cb089a882b" volumeName="kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-cni-binary-copy" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576588    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" volumeName="kubernetes.io/secret/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-serving-cert" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576596    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da622b12-25ac-43dc-8fbb-c9a974aa9f9d" volumeName="kubernetes.io/configmap/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-config" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576607    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3dbcf1c-4684-46e8-b806-1df38410a5c9" volumeName="kubernetes.io/projected/e3dbcf1c-4684-46e8-b806-1df38410a5c9-kube-api-access-nj857" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576616    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66a8e707-3e0b-42f2-988f-517d834a869c" volumeName="kubernetes.io/configmap/66a8e707-3e0b-42f2-988f-517d834a869c-serviceca" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576624    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c8687d-b9da-46b2-85ce-52cd8f6ac1dd" volumeName="kubernetes.io/configmap/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-config-volume" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576636    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c8687d-b9da-46b2-85ce-52cd8f6ac1dd" volumeName="kubernetes.io/empty-dir/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-tmp-dir" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576645    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" volumeName="kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 16:02:18.576794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576656    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" volumeName="kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path" seLinuxMountContext=""
Nov 03 16:02:18.577952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.577543    4964 factory.go:55] Registering systemd factory
Nov 03 16:02:18.577952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.577564    4964 factory.go:223] Registration of the systemd container factory successfully
Nov 03 16:02:18.577952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.577914    4964 factory.go:153] Registering CRI-O factory
Nov 03 16:02:18.577952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.577933    4964 factory.go:223] Registration of the crio container factory successfully
Nov 03 16:02:18.578155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.578011    4964 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 16:02:18.578155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.578033    4964 factory.go:103] Registering Raw factory
Nov 03 16:02:18.578155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.578048    4964 manager.go:1199] Started watching for new ooms in manager
Nov 03 16:02:18.578435 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.578421    4964 manager.go:319] Starting recovery of all containers
Nov 03 16:02:18.579140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.576664    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg" seLinuxMountContext=""
Nov 03 16:02:18.579179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579161    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 16:02:18.579209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579187    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c0598b6-8a65-4e35-9347-907aa42e8781" volumeName="kubernetes.io/projected/1c0598b6-8a65-4e35-9347-907aa42e8781-kube-api-access-2l9ch" seLinuxMountContext=""
Nov 03 16:02:18.579209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579202    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="646f7db2-f965-4cfc-883c-646eaf2975e5" volumeName="kubernetes.io/secret/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-key" seLinuxMountContext=""
Nov 03 16:02:18.579273 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579216    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a71be33d-a6e5-4b3a-b703-64cc2f214f5b" volumeName="kubernetes.io/projected/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-kube-api-access-rw48k" seLinuxMountContext=""
Nov 03 16:02:18.579273 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579231    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 16:02:18.579335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579275    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ca2e7f-7ccb-4b0e-a735-21e7d20791d7" volumeName="kubernetes.io/configmap/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-iptables-alerter-script" seLinuxMountContext=""
Nov 03 16:02:18.579335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579290    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ca2e7f-7ccb-4b0e-a735-21e7d20791d7" volumeName="kubernetes.io/projected/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-kube-api-access-kl27k" seLinuxMountContext=""
Nov 03 16:02:18.579335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579305    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b903185c-6b3b-45dd-aac2-caaa87d0b075" volumeName="kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-tuned" seLinuxMountContext=""
Nov 03 16:02:18.579335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579320    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" volumeName="kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-tmp" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579339    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579351    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22b39777-54ae-4a95-91bb-81943254f1aa" volumeName="kubernetes.io/projected/22b39777-54ae-4a95-91bb-81943254f1aa-kube-api-access-rr7g4" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579364    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2351ca8a-ffba-4733-8413-2ae0b585b489" volumeName="kubernetes.io/projected/2351ca8a-ffba-4733-8413-2ae0b585b489-kube-api-access-6x7tb" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579375    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4beaa779-098d-4007-88f8-25796f1758f0" volumeName="kubernetes.io/empty-dir/4beaa779-098d-4007-88f8-25796f1758f0-data-volume" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579402    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a78aeb2f-1c43-48f7-8b3e-4f0f9a795198" volumeName="kubernetes.io/projected/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kube-api-access-gmrnn" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579414    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" volumeName="kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579455    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b592ced-7c65-40a8-94d7-cad2c29fe081" volumeName="kubernetes.io/secret/1b592ced-7c65-40a8-94d7-cad2c29fe081-serving-cert" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579480    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e92347-295e-4c8e-bef8-519fa58d4e87" volumeName="kubernetes.io/projected/a2e92347-295e-4c8e-bef8-519fa58d4e87-kube-api-access-n8qbn" seLinuxMountContext=""
Nov 03 16:02:18.579495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579496    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a71be33d-a6e5-4b3a-b703-64cc2f214f5b" volumeName="kubernetes.io/configmap/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-config" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579510    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a6ba52-20d8-40fe-9723-54418819a3ab" volumeName="kubernetes.io/configmap/d7a6ba52-20d8-40fe-9723-54418819a3ab-telemetry-config" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579524    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17eca28e-8ea1-4193-92c1-32dd16b78e0d" volumeName="kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579537    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2351ca8a-ffba-4733-8413-2ae0b585b489" volumeName="kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-default-certificate" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579552    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2351ca8a-ffba-4733-8413-2ae0b585b489" volumeName="kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-stats-auth" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579566    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315b5fc8-0084-4395-96a5-8ae4e210b974" volumeName="kubernetes.io/projected/315b5fc8-0084-4395-96a5-8ae4e210b974-kube-api-access-vkzkr" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579581    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c8687d-b9da-46b2-85ce-52cd8f6ac1dd" volumeName="kubernetes.io/projected/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-kube-api-access-b2v2p" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579594    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ab65c33-a23d-48f3-ba16-212af80c62cb" volumeName="kubernetes.io/configmap/6ab65c33-a23d-48f3-ba16-212af80c62cb-nginx-conf" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579607    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" volumeName="kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-service-ca-bundle" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579621    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66a8e707-3e0b-42f2-988f-517d834a869c" volumeName="kubernetes.io/projected/66a8e707-3e0b-42f2-988f-517d834a869c-kube-api-access-drxvl" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579638    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d327c29c-36d3-4d01-8cab-bf10a76dbf48" volumeName="kubernetes.io/projected/d327c29c-36d3-4d01-8cab-bf10a76dbf48-kube-api-access-kvhp2" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579652    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df53f549-e19a-4304-8387-4afa8528b975" volumeName="kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579666    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3bb7151-48a1-4b4d-9784-31da848ac4f0" volumeName="kubernetes.io/configmap/f3bb7151-48a1-4b4d-9784-31da848ac4f0-konnectivity-ca" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579681    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" volumeName="kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579694    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4beaa779-098d-4007-88f8-25796f1758f0" volumeName="kubernetes.io/projected/4beaa779-098d-4007-88f8-25796f1758f0-kube-api-access-kw5tr" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579708    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e92347-295e-4c8e-bef8-519fa58d4e87" volumeName="kubernetes.io/configmap/a2e92347-295e-4c8e-bef8-519fa58d4e87-config" seLinuxMountContext=""
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579720    4964 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 16:02:18.579881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.579728    4964 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 16:02:18.581415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.580604    4964 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:02:18.581415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.580802    4964 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:02:18.583744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.583715    4964 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 16:02:18.588688 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.588662    4964 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 16:02:18.600757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.600721    4964 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 16:02:18.600757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.600752    4964 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 16:02:18.600986 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.600771    4964 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 16:02:18.600986 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.600807    4964 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 16:02:18.604317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.604281    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/1.log"
Nov 03 16:02:18.604931 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.604894    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/1.log"
Nov 03 16:02:18.605031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.604989    4964 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:02:18.609626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.609566    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/1.log"
Nov 03 16:02:18.609626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.609599    4964 generic.go:356] "Generic (PLEG): container finished" podID="1b592ced-7c65-40a8-94d7-cad2c29fe081" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91" exitCode=255
Nov 03 16:02:18.626021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.625978    4964 generic.go:356] "Generic (PLEG): container finished" podID="2e0d794832c64c25bfb1086aaf2b1f2b" containerID="fcf1749bd1b9b6d4d0181faa067aa7bffb36e026b29ec797f2202cc6af742cbc" exitCode=0
Nov 03 16:02:18.634918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.634890    4964 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:02:18.647229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.647195    4964 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="4eec39c758454315e809b597c758aa5bd8a521b17e3d60e4e50a0eae40cd66e5" exitCode=0
Nov 03 16:02:18.647229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.647222    4964 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="1e50f90380d10007cef412f977459a7d175945e1d413e90fea0f6d95d4094ad2" exitCode=0
Nov 03 16:02:18.647229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.647229    4964 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="44ee9540d273aae06022255f849efa3cf94c1405a077d6db107c6b9971ad46ae" exitCode=0
Nov 03 16:02:18.647229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.647234    4964 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="a7beef7c73935651aeb41c187755117f3904d566df7051dc05080eeb1cf7854a" exitCode=0
Nov 03 16:02:18.647229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.647240    4964 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="e17507f450838d60c3e71816609193023e4ed8b4b3104c473041a0b05837772e" exitCode=0
Nov 03 16:02:18.647554 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.647245    4964 generic.go:356] "Generic (PLEG): container finished" podID="fe4abe85-5418-426a-bb01-57b63c1f5c52" containerID="e8736a45f87f48e4fc5f162c19fd5195971f204c9917e88c9dd2901ebd3eb8e2" exitCode=0
Nov 03 16:02:18.675022 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.674995    4964 manager.go:324] Recovery completed
Nov 03 16:02:18.697549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697517    4964 cpu_manager.go:226] "Starting" policy="none"
Nov 03 16:02:18.697549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697541    4964 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 16:02:18.697718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697569    4964 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 16:02:18.697772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697748    4964 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 16:02:18.697803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697764    4964 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 16:02:18.697831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697804    4964 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 16:02:18.697831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697813    4964 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 16:02:18.697831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697821    4964 policy_none.go:50] "Start"
Nov 03 16:02:18.697831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697831    4964 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 16:02:18.697933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697843    4964 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 16:02:18.697993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697982    4964 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 16:02:18.698026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697993    4964 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 16:02:18.698026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.697999    4964 policy_none.go:44] "Start"
Nov 03 16:02:18.701081 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.701055    4964 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 16:02:18.702649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.702622    4964 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 16:02:18.705325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.705307    4964 manager.go:341] "Starting Device Plugin manager"
Nov 03 16:02:18.705919 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.705902    4964 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 16:02:18.705993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.705920    4964 server.go:87] "Starting device plugin registration server"
Nov 03 16:02:18.706183 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.706173    4964 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 16:02:18.706229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.706187    4964 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 16:02:18.706340 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.706289    4964 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 16:02:18.707719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.707527    4964 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 16:02:18.707719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.707589    4964 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 16:02:18.707719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.707597    4964 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 16:02:18.707719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.707651    4964 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:02:18.707569427Z","UUID":"6b1c07a9-ea28-4c47-8c2f-dbb85d3d3749","Handler":null,"Name":"","Endpoint":""}
Nov 03 16:02:18.707911 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.707816    4964 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 16:02:18.709035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.709018    4964 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 16:02:18.709146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.709043    4964 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 16:02:18.806354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.806319    4964 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:02:18.807758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.807737    4964 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:02:18.807867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.807774    4964 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:02:18.807867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.807787    4964 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-129.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:02:18.807867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.807841    4964 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.820834 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.820806    4964 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.820955 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.820893    4964 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.901829 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.901718    4964 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"]
Nov 03 16:02:18.902495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902445    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerStarted","Data":"002978c047f969738204c2315c4efe492f640f899f78941f6ea3e1daf0332f05"}
Nov 03 16:02:18.902495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902491    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.902645 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902518    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.902645 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902517    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerDied","Data":"fcf1749bd1b9b6d4d0181faa067aa7bffb36e026b29ec797f2202cc6af742cbc"}
Nov 03 16:02:18.902645 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902625    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"2e0d794832c64c25bfb1086aaf2b1f2b","Type":"ContainerStarted","Data":"e3342d07601bbfae8b3a015586a1cd66264ff2fd2eabfb6826fc0755a8a4ac69"}
Nov 03 16:02:18.902785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902711    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"19bdd4315736b0c6d37ff28999a14800","Type":"ContainerStarted","Data":"7010e745dc679c9b46a220cc59e1e8b31c52be781d588332f7a9e2e2dd6601c0"}
Nov 03 16:02:18.902785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.902732    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal" event={"ID":"19bdd4315736b0c6d37ff28999a14800","Type":"ContainerStarted","Data":"97cf8c2a7947eaa00b8e703b38c774a430a194faf88261b7f93b70db7fe55a4f"}
Nov 03 16:02:18.919200 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.919172    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:02:18.919293 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.919230    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.919794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.919773    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:02:18.919832 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:18.919820    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.990669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.990634    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.990669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.990668    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:18.990847 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:18.990685    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19bdd4315736b0c6d37ff28999a14800-config\") pod \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"19bdd4315736b0c6d37ff28999a14800\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.091407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.091349    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19bdd4315736b0c6d37ff28999a14800-config\") pod \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"19bdd4315736b0c6d37ff28999a14800\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.091563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.091431    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.091563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.091453    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.091563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.091362    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19bdd4315736b0c6d37ff28999a14800-config\") pod \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"19bdd4315736b0c6d37ff28999a14800\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.091563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.091510    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.091563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.091533    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0d794832c64c25bfb1086aaf2b1f2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" (UID: \"2e0d794832c64c25bfb1086aaf2b1f2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.556925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.556888    4964 apiserver.go:52] "Watching apiserver"
Nov 03 16:02:19.571605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.571570    4964 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 16:02:19.572254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.572227    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-5wvsc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal","openshift-image-registry/image-registry-6d76ff499b-x7d7l","openshift-dns/dns-default-b4wvw","openshift-image-registry/image-registry-7b94f5bc97-w9xjm","openshift-multus/multus-hr4sq","openshift-insights/insights-runtime-extractor-bcvq5","kube-system/kas-connection-checker-685895c7c9-vsbds","kube-system/kas-connection-checker-685895c7c9-wlfxh","kube-system/kas-connection-checker-685895c7c9-xw2n8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4","openshift-insights/periodic-gathering-b4vvv-lcf5m","openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4","kube-system/global-pull-secret-syncer-rtlt9","openshift-ingress-canary/ingress-canary-8qq6k","openshift-ovn-kubernetes/ovnkube-node-h45wd","openshift-network-console/networking-console-plugin-86c478bb88-8bw8n","openshift-network-operator/iptables-alerter-wblql","openshift-service-ca/service-ca-5fffc97bdf-vpqq9","openshift-ingress/router-default-859db4d5f7-xhncb","kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-njvgc","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr","openshift-console-operator/console-operator-7cd56b8dbc-4ftv2","openshift-multus/multus-additional-cni-plugins-zpddq","openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t","kube-system/konnectivity-agent-r59q4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q","openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb","openshift-dns/node-resolver-mlccd","openshift-network-diagnostics/network-check-target-kdkkw","openshift-cluster-node-tuning-operator/tuned-ww48r","openshift-insights/insights-operator-7c4986d449-vrzhj"]
Nov 03 16:02:19.575788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.575739    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.577598 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.577573    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.577961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.577941    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.578596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.578581    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.579330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.579314    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:19.580902 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.580881    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.580993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.580886    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 16:02:19.580993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.580886    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.580993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.580889    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.581835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.581807    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 16:02:19.582241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.582220    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 16:02:19.582317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.582242    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 16:02:19.582317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.582223    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.582596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.582579    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 16:02:19.582725 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.582709    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.582870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.582847    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5c5gf\""
Nov 03 16:02:19.583785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.583770    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:19.585124 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.585108    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.585537 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.585523    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 16:02:19.586115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.586101    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.586196 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.586180    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.586688 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.586669    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 16:02:19.586783 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.586736    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 16:02:19.587006 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.586984    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 16:02:19.587697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.587677    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.590772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.590746    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.597367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.597339    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 16:02:19.597475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.597417    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.609316 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.609287    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.611327 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.611307    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 16:02:19.614247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.614222    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.630082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630048    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.630082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630073    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.630276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630111    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.630276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630177    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.630276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630203    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.630480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630465    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 16:02:19.630570 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630557    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.630606 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630572    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 16:02:19.630643 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.630626    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 16:02:19.631048 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631036    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.631188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631160    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.631188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631172    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 16:02:19.631188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631163    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.631354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631212    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 16:02:19.631354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631226    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-ccc6l\""
Nov 03 16:02:19.631354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631233    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 16:02:19.631354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631338    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.631354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631342    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.631529 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631349    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 16:02:19.631529 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631447    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.631767 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631752    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.631831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631810    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.631831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631811    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.631917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631826    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.631917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631845    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 16:02:19.631917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631846    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.631917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631869    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 16:02:19.631917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631875    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 16:02:19.631917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.631880    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 16:02:19.632707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632695    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.632755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632719    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 16:02:19.632755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632719    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 16:02:19.632755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632734    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.632755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632750    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.632894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632763    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 16:02:19.632894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632772    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.632894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.632788    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.633158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633145    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 16:02:19.633190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633182    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 16:02:19.633219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633211    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.633301 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633288    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.633381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633369    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.633443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633396    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:02:19.633443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633435    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 16:02:19.633516 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.633479    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 16:02:19.634047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634036    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.634167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634138    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 16:02:19.634167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634152    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 16:02:19.634167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634160    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 16:02:19.634456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634438    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634527    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634669    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634677    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634684    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c56qh\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634680    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5rhh2\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634705    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 16:02:19.634731 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634712    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634758    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r589\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634758    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634793    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634800    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dchj7\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634804    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qk9pq\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634818    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jdvsk\""
Nov 03 16:02:19.634999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.634824    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2h492\""
Nov 03 16:02:19.635458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635446    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgwbg\""
Nov 03 16:02:19.635563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635478    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\""
Nov 03 16:02:19.635563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635492    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drdlv\""
Nov 03 16:02:19.635563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635485    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbm9f\""
Nov 03 16:02:19.635563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635560    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jk68z\""
Nov 03 16:02:19.635873 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635855    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8szjr\""
Nov 03 16:02:19.635966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635911    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dfrrd\""
Nov 03 16:02:19.635966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635911    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\""
Nov 03 16:02:19.635966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635931    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 16:02:19.636070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.635975    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 16:02:19.636161 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.636149    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f7qmg\""
Nov 03 16:02:19.636209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.636148    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6rcnt\""
Nov 03 16:02:19.636269 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.636256    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bb2h\""
Nov 03 16:02:19.639114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.639101    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:02:19.639167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.639123    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 16:02:19.639167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.639121    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 16:02:19.643025 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.643007    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 16:02:19.643312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.643289    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 16:02:19.643350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.643338    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 16:02:19.659432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.659404    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.659577 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.659563    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.673917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.673889    4964 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 16:02:19.676576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.676552    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:02:19.676706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.676599    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:02:19.676706 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.676613    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.676706 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.676650    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal"
Nov 03 16:02:19.695399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695362    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-k8s-cni-cncf-io\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.695575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695419    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-host\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.695575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695440    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5tr\" (UniqueName: \"kubernetes.io/projected/4beaa779-098d-4007-88f8-25796f1758f0-kube-api-access-kw5tr\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.695575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695523    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-tmp-dir\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.695575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695565    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-default-certificate\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695597    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-ovn\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695624    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-cnibin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695648    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysconfig\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695660    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-tmp-dir\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695691    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-sys\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695711    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.695778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695743    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-systemd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695826    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-default-certificate\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695831    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-etc-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695866    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695908    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkzkr\" (UniqueName: \"kubernetes.io/projected/315b5fc8-0084-4395-96a5-8ae4e210b974-kube-api-access-vkzkr\") pod \"kas-connection-checker-685895c7c9-xw2n8\" (UID: \"315b5fc8-0084-4395-96a5-8ae4e210b974\") " pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695930    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66a8e707-3e0b-42f2-988f-517d834a869c-serviceca\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695955    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3387caf-55fb-4a86-9b89-f38c8fbe1915-original-pull-secret\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695958    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-hostroot\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.695999    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696027    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-kubernetes\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696058    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5lp\" (UniqueName: \"kubernetes.io/projected/1b592ced-7c65-40a8-94d7-cad2c29fe081-kube-api-access-cr5lp\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696085    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.696118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696109    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-netns\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696139    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696147    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-bin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696187    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696213    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9v5s\" (UniqueName: \"kubernetes.io/projected/a467d9db-7d44-47e3-a142-30cb089a882b-kube-api-access-k9v5s\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696256    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66a8e707-3e0b-42f2-988f-517d834a869c-serviceca\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696269    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxw4\" (UniqueName: \"kubernetes.io/projected/646f7db2-f965-4cfc-883c-646eaf2975e5-kube-api-access-2pxw4\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696303    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696322    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-tmp\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696338    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhp2\" (UniqueName: \"kubernetes.io/projected/d327c29c-36d3-4d01-8cab-bf10a76dbf48-kube-api-access-kvhp2\") pod \"volume-data-source-validator-58c9f5f46d-qfwp4\" (UID: \"d327c29c-36d3-4d01-8cab-bf10a76dbf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-tmp\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696513    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696542    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngjp\" (UniqueName: \"kubernetes.io/projected/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2-kube-api-access-kngjp\") pod \"kas-connection-checker-685895c7c9-wlfxh\" (UID: \"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2\") " pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696557    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-dbus\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696597    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7tb\" (UniqueName: \"kubernetes.io/projected/2351ca8a-ffba-4733-8413-2ae0b585b489-kube-api-access-6x7tb\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696614    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-multus-daemon-config\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696662    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-cabundle\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696694    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696698    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696735    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696756    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696774    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74rn\" (UniqueName: \"kubernetes.io/projected/fe4abe85-5418-426a-bb01-57b63c1f5c52-kube-api-access-g74rn\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696791    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3bb7151-48a1-4b4d-9784-31da848ac4f0-agent-certs\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:02:19.696793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696811    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxvl\" (UniqueName: \"kubernetes.io/projected/66a8e707-3e0b-42f2-988f-517d834a869c-kube-api-access-drxvl\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696831    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-tmp\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696848    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvlj\" (UniqueName: \"kubernetes.io/projected/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-kube-api-access-6vvlj\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696863    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696879    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e92347-295e-4c8e-bef8-519fa58d4e87-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696885    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-cabundle\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696886    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696907    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-multus\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696959    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-multus-certs\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696982    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-netns\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696981    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-multus-daemon-config\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.696998    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697014    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697029    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-sys-fs\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697152    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697171    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697202    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e92347-295e-4c8e-bef8-519fa58d4e87-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697213    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697261    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-tmp\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697295    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl27k\" (UniqueName: \"kubernetes.io/projected/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-kube-api-access-kl27k\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697332    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-etc-kubernetes\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697367    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3bb7151-48a1-4b4d-9784-31da848ac4f0-agent-certs\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697373    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw48k\" (UniqueName: \"kubernetes.io/projected/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-kube-api-access-rw48k\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697445    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.697511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697493    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b592ced-7c65-40a8-94d7-cad2c29fe081-serving-cert\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697537    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-registration-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697613    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-device-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697648    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-serving-cert\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697667    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-config\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697674    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b592ced-7c65-40a8-94d7-cad2c29fe081-serving-cert\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697701    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-var-lib-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697798    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-env-overrides\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697814    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-config\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697846    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklmt\" (UniqueName: \"kubernetes.io/projected/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-kube-api-access-dklmt\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697878    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697877    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-serving-cert\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.697947    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4vs\" (UniqueName: \"kubernetes.io/projected/6a30fac5-ac40-4179-9628-fa3203ddb77b-kube-api-access-vw4vs\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698033    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab65c33-a23d-48f3-ba16-212af80c62cb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698045    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-env-overrides\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698126    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698145    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-key\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:19.698260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698255    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-log-socket\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698284    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698332    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab65c33-a23d-48f3-ba16-212af80c62cb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698402    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-snapshots\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698438    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/646f7db2-f965-4cfc-883c-646eaf2975e5-signing-key\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698467    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-snapshots\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698470    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698562    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9ch\" (UniqueName: \"kubernetes.io/projected/1c0598b6-8a65-4e35-9347-907aa42e8781-kube-api-access-2l9ch\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698607    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-conf-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698625    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698640    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-conf\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698660    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djljv\" (UniqueName: \"kubernetes.io/projected/b903185c-6b3b-45dd-aac2-caaa87d0b075-kube-api-access-djljv\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-trusted-ca\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698759    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.698781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698778    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698797    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698832    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.698873    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qbn\" (UniqueName: \"kubernetes.io/projected/a2e92347-295e-4c8e-bef8-519fa58d4e87-kube-api-access-n8qbn\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699005    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-cni-binary-copy\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699032    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699057    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-tuned\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699090    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4beaa779-098d-4007-88f8-25796f1758f0-crio-socket\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699112    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovn-node-metrics-cert\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699128    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-tuned\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699128    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-cnibin\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699180    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2v2p\" (UniqueName: \"kubernetes.io/projected/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-kube-api-access-b2v2p\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699191    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a467d9db-7d44-47e3-a142-30cb089a882b-cni-binary-copy\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699191    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-trusted-ca\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699220    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-systemd-units\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699233    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699245    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-systemd\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699427    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699449    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699454    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovn-node-metrics-cert\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699485    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:19.699508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699507    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699524    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66a8e707-3e0b-42f2-988f-517d834a869c-host\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699570    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-bin\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699618    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699657    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699667    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-kubelet\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-script-lib\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699743    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4abe85-5418-426a-bb01-57b63c1f5c52-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699788    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-system-cni-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699788    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699837    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699864    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699912    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699918    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-script-lib\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.699942    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-config-volume\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700044    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700073    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-serving-cert\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700108    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700158    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700198    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700213    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-config-volume\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.700399 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700299    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700377    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700432    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700465    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-node-log\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700493    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj857\" (UniqueName: \"kubernetes.io/projected/e3dbcf1c-4684-46e8-b806-1df38410a5c9-kube-api-access-nj857\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700520    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30fac5-ac40-4179-9628-fa3203ddb77b-tmp-dir\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700541    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-serving-cert\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700548    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kube-api-access-gmrnn\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700577    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700576    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4zf\" (UniqueName: \"kubernetes.io/projected/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-kube-api-access-jq4zf\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700655    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30fac5-ac40-4179-9628-fa3203ddb77b-tmp-dir\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700740    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7g4\" (UniqueName: \"kubernetes.io/projected/22b39777-54ae-4a95-91bb-81943254f1aa-kube-api-access-rr7g4\") pod \"network-check-source-6f78d59f99-mhc9q\" (UID: \"22b39777-54ae-4a95-91bb-81943254f1aa\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700803    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-netd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700796    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700828    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-config\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700879    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700933    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a6ba52-20d8-40fe-9723-54418819a3ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700978    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-kubelet\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.700995    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b592ced-7c65-40a8-94d7-cad2c29fe081-config\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701057    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-socket-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701095    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701130    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dp9\" (UniqueName: \"kubernetes.io/projected/d7a6ba52-20d8-40fe-9723-54418819a3ab-kube-api-access-w5dp9\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701181    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a6ba52-20d8-40fe-9723-54418819a3ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701228    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-modprobe-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701252    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-kubelet-config\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701276    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.701294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701307    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701308    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-system-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701421    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-run\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701444    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4beaa779-098d-4007-88f8-25796f1758f0-data-volume\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701460    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701476    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-slash\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701525    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4beaa779-098d-4007-88f8-25796f1758f0-data-volume\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701529    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dbcf1c-4684-46e8-b806-1df38410a5c9-cert\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701637    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a30fac5-ac40-4179-9628-fa3203ddb77b-hosts-file\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701675    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701710    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701730    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dbcf1c-4684-46e8-b806-1df38410a5c9-cert\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701732    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-iptables-alerter-script\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701776    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-host-slash\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701800    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-stats-auth\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701815    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701848    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-lib-modules\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701942    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.701955    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-iptables-alerter-script\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702014    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702013    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702033    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-stats-auth\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.702104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702041    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3bb7151-48a1-4b4d-9784-31da848ac4f0-konnectivity-ca\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702138    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jl7\" (UniqueName: \"kubernetes.io/projected/dfe05141-01c7-4446-a5b5-e61abe65dfa2-kube-api-access-w8jl7\") pod \"kas-connection-checker-685895c7c9-vsbds\" (UID: \"dfe05141-01c7-4446-a5b5-e61abe65dfa2\") " pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702227    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702245    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e92347-295e-4c8e-bef8-519fa58d4e87-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702279    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3bb7151-48a1-4b4d-9784-31da848ac4f0-konnectivity-ca\") pod \"konnectivity-agent-r59q4\" (UID: \"f3bb7151-48a1-4b4d-9784-31da848ac4f0\") " pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702290    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-socket-dir-parent\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702314    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-config\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702331    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-var-lib-kubelet\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702550    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-ovnkube-config\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702576    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e92347-295e-4c8e-bef8-519fa58d4e87-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:19.702827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.702730    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.711420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.711372    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.711672 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.711646    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5lp\" (UniqueName: \"kubernetes.io/projected/1b592ced-7c65-40a8-94d7-cad2c29fe081-kube-api-access-cr5lp\") pod \"console-operator-7cd56b8dbc-4ftv2\" (UID: \"1b592ced-7c65-40a8-94d7-cad2c29fe081\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:19.711756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.711646    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxw4\" (UniqueName: \"kubernetes.io/projected/646f7db2-f965-4cfc-883c-646eaf2975e5-kube-api-access-2pxw4\") pod \"service-ca-5fffc97bdf-vpqq9\" (UID: \"646f7db2-f965-4cfc-883c-646eaf2975e5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vpqq9"
Nov 03 16:02:19.712105 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.712089    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9v5s\" (UniqueName: \"kubernetes.io/projected/a467d9db-7d44-47e3-a142-30cb089a882b-kube-api-access-k9v5s\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.712149 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.712134    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl27k\" (UniqueName: \"kubernetes.io/projected/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-kube-api-access-kl27k\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.712780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.712764    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxvl\" (UniqueName: \"kubernetes.io/projected/66a8e707-3e0b-42f2-988f-517d834a869c-kube-api-access-drxvl\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.712948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.712933    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7tb\" (UniqueName: \"kubernetes.io/projected/2351ca8a-ffba-4733-8413-2ae0b585b489-kube-api-access-6x7tb\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.713005 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.712990    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvlj\" (UniqueName: \"kubernetes.io/projected/bb4f4939-d15b-4b06-bc6e-27ca8b2039ec-kube-api-access-6vvlj\") pod \"insights-operator-7c4986d449-vrzhj\" (UID: \"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec\") " pod="openshift-insights/insights-operator-7c4986d449-vrzhj"
Nov 03 16:02:19.713995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.713981    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5tr\" (UniqueName: \"kubernetes.io/projected/4beaa779-098d-4007-88f8-25796f1758f0-kube-api-access-kw5tr\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.714195 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.714170    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkzkr\" (UniqueName: \"kubernetes.io/projected/315b5fc8-0084-4395-96a5-8ae4e210b974-kube-api-access-vkzkr\") pod \"kas-connection-checker-685895c7c9-xw2n8\" (UID: \"315b5fc8-0084-4395-96a5-8ae4e210b974\") " pod="kube-system/kas-connection-checker-685895c7c9-xw2n8"
Nov 03 16:02:19.714367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.714354    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhp2\" (UniqueName: \"kubernetes.io/projected/d327c29c-36d3-4d01-8cab-bf10a76dbf48-kube-api-access-kvhp2\") pod \"volume-data-source-validator-58c9f5f46d-qfwp4\" (UID: \"d327c29c-36d3-4d01-8cab-bf10a76dbf48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qfwp4"
Nov 03 16:02:19.714608 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.714592    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74rn\" (UniqueName: \"kubernetes.io/projected/fe4abe85-5418-426a-bb01-57b63c1f5c52-kube-api-access-g74rn\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.720505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.720480    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.721448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.721424    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9ch\" (UniqueName: \"kubernetes.io/projected/1c0598b6-8a65-4e35-9347-907aa42e8781-kube-api-access-2l9ch\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:19.721540 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.721500    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qbn\" (UniqueName: \"kubernetes.io/projected/a2e92347-295e-4c8e-bef8-519fa58d4e87-kube-api-access-n8qbn\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f849q\" (UID: \"a2e92347-295e-4c8e-bef8-519fa58d4e87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q"
Nov 03 16:02:19.721608 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.721556    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngjp\" (UniqueName: \"kubernetes.io/projected/fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2-kube-api-access-kngjp\") pod \"kas-connection-checker-685895c7c9-wlfxh\" (UID: \"fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2\") " pod="kube-system/kas-connection-checker-685895c7c9-wlfxh"
Nov 03 16:02:19.721696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.721679    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4vs\" (UniqueName: \"kubernetes.io/projected/6a30fac5-ac40-4179-9628-fa3203ddb77b-kube-api-access-vw4vs\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.721775 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.721759    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw48k\" (UniqueName: \"kubernetes.io/projected/a71be33d-a6e5-4b3a-b703-64cc2f214f5b-kube-api-access-rw48k\") pod \"service-ca-operator-544d749c8c-9sxgb\" (UID: \"a71be33d-a6e5-4b3a-b703-64cc2f214f5b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb"
Nov 03 16:02:19.721807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.721770    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djljv\" (UniqueName: \"kubernetes.io/projected/b903185c-6b3b-45dd-aac2-caaa87d0b075-kube-api-access-djljv\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.722080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.722066    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklmt\" (UniqueName: \"kubernetes.io/projected/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-kube-api-access-dklmt\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.728370 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.728346    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kube-api-access-gmrnn\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.728505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.728417    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dp9\" (UniqueName: \"kubernetes.io/projected/d7a6ba52-20d8-40fe-9723-54418819a3ab-kube-api-access-w5dp9\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.728505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.728430    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jl7\" (UniqueName: \"kubernetes.io/projected/dfe05141-01c7-4446-a5b5-e61abe65dfa2-kube-api-access-w8jl7\") pod \"kas-connection-checker-685895c7c9-vsbds\" (UID: \"dfe05141-01c7-4446-a5b5-e61abe65dfa2\") " pod="kube-system/kas-connection-checker-685895c7c9-vsbds"
Nov 03 16:02:19.728574 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.728563    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.729162 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.729139    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7g4\" (UniqueName: \"kubernetes.io/projected/22b39777-54ae-4a95-91bb-81943254f1aa-kube-api-access-rr7g4\") pod \"network-check-source-6f78d59f99-mhc9q\" (UID: \"22b39777-54ae-4a95-91bb-81943254f1aa\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mhc9q"
Nov 03 16:02:19.733776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.733746    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") pod \"periodic-gathering-b4vvv-lcf5m\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") " pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:02:19.733776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.733773    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4zf\" (UniqueName: \"kubernetes.io/projected/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-kube-api-access-jq4zf\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.733941 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.733793    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8262d\" (UniqueName: \"kubernetes.io/projected/e3c566eb-b54b-47e7-a848-c1cacacc63e7-kube-api-access-8262d\") pod \"network-check-target-kdkkw\" (UID: \"e3c566eb-b54b-47e7-a848-c1cacacc63e7\") " pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:19.733941 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.733918    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.734024 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.733974    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj857\" (UniqueName: \"kubernetes.io/projected/e3dbcf1c-4684-46e8-b806-1df38410a5c9-kube-api-access-nj857\") pod \"ingress-canary-8qq6k\" (UID: \"e3dbcf1c-4684-46e8-b806-1df38410a5c9\") " pod="openshift-ingress-canary/ingress-canary-8qq6k"
Nov 03 16:02:19.739407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.739360    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2v2p\" (UniqueName: \"kubernetes.io/projected/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-kube-api-access-b2v2p\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.802814 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802773    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a30fac5-ac40-4179-9628-fa3203ddb77b-hosts-file\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.802814 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802814    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802835    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-host-slash\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802891    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-lib-modules\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802914    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a30fac5-ac40-4179-9628-fa3203ddb77b-hosts-file\") pod \"node-resolver-mlccd\" (UID: \"6a30fac5-ac40-4179-9628-fa3203ddb77b\") " pod="openshift-dns/node-resolver-mlccd"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802921    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-socket-dir-parent\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802921    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64ca2e7f-7ccb-4b0e-a735-21e7d20791d7-host-slash\") pod \"iptables-alerter-wblql\" (UID: \"64ca2e7f-7ccb-4b0e-a735-21e7d20791d7\") " pod="openshift-network-operator/iptables-alerter-wblql"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802962    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-var-lib-kubelet\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.802980    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-socket-dir-parent\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803034    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-lib-modules\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803034    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-var-lib-kubelet\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803035    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-k8s-cni-cncf-io\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803074    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-k8s-cni-cncf-io\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803104    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-host\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803132    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-ovn\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803148    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-host\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803157    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-cnibin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803194    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-ovn\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803215    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-cnibin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803224    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysconfig\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803243    4964 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803261    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysconfig\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803254    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-sys\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803279    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-sys\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803291    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-systemd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803313    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-etc-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803330    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803345    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-systemd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803364    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-hostroot\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803364    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-etc-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803399    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803406    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-kubernetes\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803422    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-hostroot\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803432    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-kubernetes\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803432    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.803595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803561    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-netns\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803617    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-netns\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803613    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-bin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803669    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803705    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803713    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803732    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-dbus\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803666    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-bin\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803774    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-multus\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803799    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-dbus\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803803    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-cni-multus\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803818    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-multus-certs\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-netns\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803881    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803920    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-netns\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803927    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-run-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803932    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-run-multus-certs\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803951    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803982    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-sys-fs\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.803988    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804007    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804033    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-etc-kubernetes\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804054    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-sys-fs\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804080    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-etc-kubernetes\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804114    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.804139    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804183    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-registration-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804228    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-device-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804283    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-var-lib-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804289    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-registration-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804315    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-log-socket\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804350    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804429    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-log-socket\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.804447    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.304423245 +0000 UTC m=+1.817805020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804476    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-device-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804499    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-var-lib-openvswitch\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804512    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804618    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-conf-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.804639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804641    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-conf\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804668    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804704    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804730    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-multus-conf-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804737    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804802    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4beaa779-098d-4007-88f8-25796f1758f0-crio-socket\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804828    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-cnibin\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804859    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-systemd-units\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804865    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-conf\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804902    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-systemd\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804929    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4beaa779-098d-4007-88f8-25796f1758f0-crio-socket\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.804926    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805006    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805058    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-cnibin\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805125    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-systemd-units\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805194    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-systemd\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805405    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805432    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4beaa779-098d-4007-88f8-25796f1758f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805448    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66a8e707-3e0b-42f2-988f-517d834a869c-host\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805512    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-bin\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.805529    4964 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805541    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-kubelet\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805582    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-system-cni-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805610    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66a8e707-3e0b-42f2-988f-517d834a869c-host\") pod \"node-ca-5wvsc\" (UID: \"66a8e707-3e0b-42f2-988f-517d834a869c\") " pod="openshift-image-registry/node-ca-5wvsc"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805624    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805660    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-kubelet\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805666    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805705    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-bin\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805720    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-node-log\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805751    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-netd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805777    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805823    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-kubelet\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.805857    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle podName:2351ca8a-ffba-4733-8413-2ae0b585b489 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.305831129 +0000 UTC m=+1.819212908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle") pod "router-default-859db4d5f7-xhncb" (UID: "2351ca8a-ffba-4733-8413-2ae0b585b489") : configmap references non-existent config key: service-ca.crt
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805853    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-sysctl-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805894    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-socket-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805926    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-modprobe-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805941    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-node-log\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805952    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-kubelet-config\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805980    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-system-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806001    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-run\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.806018    4964 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806026    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-slash\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.806062    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.306047877 +0000 UTC m=+1.819429632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806101    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-cni-netd\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806112    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5dea52a-10d2-4ca8-a4e9-096f66ad73fc-host-slash\") pod \"ovnkube-node-h45wd\" (UID: \"b5dea52a-10d2-4ca8-a4e9-096f66ad73fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806154    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4abe85-5418-426a-bb01-57b63c1f5c52-system-cni-dir\") pod \"multus-additional-cni-plugins-zpddq\" (UID: \"fe4abe85-5418-426a-bb01-57b63c1f5c52\") " pod="openshift-multus/multus-additional-cni-plugins-zpddq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806178    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"image-registry-7b94f5bc97-w9xjm\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") " pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.806197    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806244    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3387caf-55fb-4a86-9b89-f38c8fbe1915-kubelet-config\") pod \"global-pull-secret-syncer-rtlt9\" (UID: \"d3387caf-55fb-4a86-9b89-f38c8fbe1915\") " pod="kube-system/global-pull-secret-syncer-rtlt9"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.806202    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs podName:1c0598b6-8a65-4e35-9347-907aa42e8781 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.306189907 +0000 UTC m=+1.819571663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs") pod "network-metrics-daemon-njvgc" (UID: "1c0598b6-8a65-4e35-9347-907aa42e8781") : secret "metrics-daemon-secret" not found
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806270    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-etc-modprobe-d\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:19.806287    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:20.306269887 +0000 UTC m=+1.819651655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806301    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-system-cni-dir\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806316    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a78aeb2f-1c43-48f7-8b3e-4f0f9a795198-socket-dir\") pod \"aws-ebs-csi-driver-node-6wkc4\" (UID: \"a78aeb2f-1c43-48f7-8b3e-4f0f9a795198\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wkc4"
Nov 03 16:02:19.806501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806341    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b903185c-6b3b-45dd-aac2-caaa87d0b075-run\") pod \"tuned-ww48r\" (UID: \"b903185c-6b3b-45dd-aac2-caaa87d0b075\") " pod="openshift-cluster-node-tuning-operator/tuned-ww48r"
Nov 03 16:02:19.808623 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806559    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"image-registry-6d76ff499b-x7d7l\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") " pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.808623 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.806686    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66c8687d-b9da-46b2-85ce-52cd8f6ac1dd-metrics-tls\") pod \"dns-default-b4wvw\" (UID: \"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd\") " pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:19.808623 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.807188    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2351ca8a-ffba-4733-8413-2ae0b585b489-metrics-certs\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:19.808623 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.805888    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a467d9db-7d44-47e3-a142-30cb089a882b-host-var-lib-kubelet\") pod \"multus-hr4sq\" (UID: \"a467d9db-7d44-47e3-a142-30cb089a882b\") " pod="openshift-multus/multus-hr4sq"
Nov 03 16:02:19.808623 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.807957    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da622b12-25ac-43dc-8fbb-c9a974aa9f9d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vq8cr\" (UID: \"da622b12-25ac-43dc-8fbb-c9a974aa9f9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.876140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.876106    4964 scope.go:122] "RemoveContainer" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91"
Nov 03 16:02:19.878155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.878129    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:19.882011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.881978    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"
Nov 03 16:02:19.886587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.886567    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:19.887932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:19.887913    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:20.188718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.188553    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d76ff499b-x7d7l"]
Nov 03 16:02:20.191299 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:20.191267    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf53f549_e19a_4304_8387_4afa8528b975.slice/crio-7b7a2b63c99569b33fec7241c9d3cbfc89dab31bc8413cdc0a924e6e79401b06 WatchSource:0}: Error finding container 7b7a2b63c99569b33fec7241c9d3cbfc89dab31bc8413cdc0a924e6e79401b06: Status 404 returned error can't find the container with id 7b7a2b63c99569b33fec7241c9d3cbfc89dab31bc8413cdc0a924e6e79401b06
Nov 03 16:02:20.220138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.220102    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr"]
Nov 03 16:02:20.263277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.263256    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:02:20.276899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.276878    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4wvw"]
Nov 03 16:02:20.279208 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:20.279172    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c8687d_b9da_46b2_85ce_52cd8f6ac1dd.slice/crio-0716fb20dafc5d7cb8f7c2d981821f34ee049102d697255edf265da4a6dee114 WatchSource:0}: Error finding container 0716fb20dafc5d7cb8f7c2d981821f34ee049102d697255edf265da4a6dee114: Status 404 returned error can't find the container with id 0716fb20dafc5d7cb8f7c2d981821f34ee049102d697255edf265da4a6dee114
Nov 03 16:02:20.296497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.296467    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b94f5bc97-w9xjm"]
Nov 03 16:02:20.299861 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:20.299830    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eca28e_8ea1_4193_92c1_32dd16b78e0d.slice/crio-96fb3d5b49793e6a5ac7508368b0bc678a6dd66a4a046aa09fbf4c346a0b3c85 WatchSource:0}: Error finding container 96fb3d5b49793e6a5ac7508368b0bc678a6dd66a4a046aa09fbf4c346a0b3c85: Status 404 returned error can't find the container with id 96fb3d5b49793e6a5ac7508368b0bc678a6dd66a4a046aa09fbf4c346a0b3c85
Nov 03 16:02:20.310778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.310714    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:20.310877 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.310796    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:20.310877 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.310828    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:20.310877 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.310868    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:20.311035 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.310897    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:20.311035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.310905    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:20.311035 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.310968    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:21.310947655 +0000 UTC m=+2.824329425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:20.311035 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.311022    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:20.311228 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.311079    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:21.311063471 +0000 UTC m=+2.824445245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:20.311228 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.311142    4964 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:20.311228 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.311178    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:21.311166917 +0000 UTC m=+2.824548687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:20.311701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.311641    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2351ca8a-ffba-4733-8413-2ae0b585b489-service-ca-bundle\") pod \"router-default-859db4d5f7-xhncb\" (UID: \"2351ca8a-ffba-4733-8413-2ae0b585b489\") " pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:20.313532 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.313511    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0598b6-8a65-4e35-9347-907aa42e8781-metrics-certs\") pod \"network-metrics-daemon-njvgc\" (UID: \"1c0598b6-8a65-4e35-9347-907aa42e8781\") " pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:20.476819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.476775    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:20.484753 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.484723    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njvgc"
Nov 03 16:02:20.667578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.667543    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr" event={"ID":"da622b12-25ac-43dc-8fbb-c9a974aa9f9d","Type":"ContainerStarted","Data":"bd61fed78a8c62a31abd9d9ff2f5a880cc5903fded6b2577722d3dff4ede704e"}
Nov 03 16:02:20.669177 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.669151    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm" event={"ID":"17eca28e-8ea1-4193-92c1-32dd16b78e0d","Type":"ContainerStarted","Data":"2923eeb96076e7d1b51c7bd922fa97d5a8c6bcbe9e8f187ca4a189d488e9df3c"}
Nov 03 16:02:20.669318 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.669199    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm" event={"ID":"17eca28e-8ea1-4193-92c1-32dd16b78e0d","Type":"ContainerStarted","Data":"96fb3d5b49793e6a5ac7508368b0bc678a6dd66a4a046aa09fbf4c346a0b3c85"}
Nov 03 16:02:20.670883 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.670861    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:02:20.671346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.671324    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/1.log"
Nov 03 16:02:20.671490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.671363    4964 generic.go:356] "Generic (PLEG): container finished" podID="1b592ced-7c65-40a8-94d7-cad2c29fe081" containerID="da0f5652214ce00301e16c11d816367d2b7d9a80ece12dad5c3d983f7f5593ed" exitCode=255
Nov 03 16:02:20.671545 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.671490    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" event={"ID":"1b592ced-7c65-40a8-94d7-cad2c29fe081","Type":"ContainerDied","Data":"da0f5652214ce00301e16c11d816367d2b7d9a80ece12dad5c3d983f7f5593ed"}
Nov 03 16:02:20.671633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.671550    4964 scope.go:122] "RemoveContainer" containerID="d15cc9bb766719183df7c0c9d0ddd2ed0e779c603b08b5c65878511821425a91"
Nov 03 16:02:20.671918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.671902    4964 scope.go:122] "RemoveContainer" containerID="da0f5652214ce00301e16c11d816367d2b7d9a80ece12dad5c3d983f7f5593ed"
Nov 03 16:02:20.672147 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:20.672125    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4ftv2_openshift-console-operator(1b592ced-7c65-40a8-94d7-cad2c29fe081)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" podUID="1b592ced-7c65-40a8-94d7-cad2c29fe081"
Nov 03 16:02:20.674330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.674299    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l" event={"ID":"df53f549-e19a-4304-8387-4afa8528b975","Type":"ContainerStarted","Data":"750d90006b95b446559274619a02c24158258f482ffa0f97a2f697ff1aac1b11"}
Nov 03 16:02:20.674434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.674356    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l" event={"ID":"df53f549-e19a-4304-8387-4afa8528b975","Type":"ContainerStarted","Data":"7b7a2b63c99569b33fec7241c9d3cbfc89dab31bc8413cdc0a924e6e79401b06"}
Nov 03 16:02:20.678059 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.677150    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4wvw" event={"ID":"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd","Type":"ContainerStarted","Data":"0716fb20dafc5d7cb8f7c2d981821f34ee049102d697255edf265da4a6dee114"}
Nov 03 16:02:20.788875 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.788833    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-njvgc"]
Nov 03 16:02:20.811594 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:20.811490    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-859db4d5f7-xhncb"]
Nov 03 16:02:20.816096 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:20.816061    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2351ca8a_ffba_4733_8413_2ae0b585b489.slice/crio-767593785f031266c53a5d23b815f3cecba4d35445474809e95ba47bf4cd24cd WatchSource:0}: Error finding container 767593785f031266c53a5d23b815f3cecba4d35445474809e95ba47bf4cd24cd: Status 404 returned error can't find the container with id 767593785f031266c53a5d23b815f3cecba4d35445474809e95ba47bf4cd24cd
Nov 03 16:02:21.323376 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.323331    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:21.323709 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.323418    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:21.323709 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.323451    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:21.323709 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:21.323590    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:21.323709 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:21.323649    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.323632272 +0000 UTC m=+4.837014044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:21.324022 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:21.324007    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:21.324114 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:21.324058    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.324043808 +0000 UTC m=+4.837425577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:21.324179 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:21.324114    4964 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:21.324179 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:21.324147    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls podName:d7a6ba52-20d8-40fe-9723-54418819a3ab nodeName:}" failed. No retries permitted until 2026-11-03 16:02:23.324137146 +0000 UTC m=+4.837518905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7ql5t" (UID: "d7a6ba52-20d8-40fe-9723-54418819a3ab") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:02:21.413591 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.413550    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:21.596123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.596034    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:21.684632 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.684601    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:02:21.687230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.686660    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-859db4d5f7-xhncb" event={"ID":"2351ca8a-ffba-4733-8413-2ae0b585b489","Type":"ContainerStarted","Data":"f796073fb3ecb1cc555a630973e8bfe976a47c6af82a35231eb4fb3160855121"}
Nov 03 16:02:21.687230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.686730    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-859db4d5f7-xhncb" event={"ID":"2351ca8a-ffba-4733-8413-2ae0b585b489","Type":"ContainerStarted","Data":"767593785f031266c53a5d23b815f3cecba4d35445474809e95ba47bf4cd24cd"}
Nov 03 16:02:21.691344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.689233    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njvgc" event={"ID":"1c0598b6-8a65-4e35-9347-907aa42e8781","Type":"ContainerStarted","Data":"4a1418a7d26fbc721b3d3867831cc0100c6c2955760b1c515874841cdec4f77a"}
Nov 03 16:02:21.691344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:21.689342    4964 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:02:22.477766 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.477732    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:22.482854 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.482654    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:22.698426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.698368    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr" event={"ID":"da622b12-25ac-43dc-8fbb-c9a974aa9f9d","Type":"ContainerStarted","Data":"9ca9d74de31eda014b0b22882b6955dfa2bfd9a0732339bb36332cb66dc05f0f"}
Nov 03 16:02:22.698941 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.698485    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vq8cr" event={"ID":"da622b12-25ac-43dc-8fbb-c9a974aa9f9d","Type":"ContainerStarted","Data":"1584c4edaa6d84395f40145142039d346d54fa3631527dab660fe03cdfd31e36"}
Nov 03 16:02:22.700347 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.700310    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njvgc" event={"ID":"1c0598b6-8a65-4e35-9347-907aa42e8781","Type":"ContainerStarted","Data":"f2c53a7d694d340d2e4f0c1d7077c800bde4a500ea804fe53c38d365749078bc"}
Nov 03 16:02:22.700518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.700372    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njvgc" event={"ID":"1c0598b6-8a65-4e35-9347-907aa42e8781","Type":"ContainerStarted","Data":"283b95d0e6bc7f5fb87bc653a9cbca37cbc068e53aea0aba7b06e10fb6146233"}
Nov 03 16:02:22.702262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.702190    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4wvw" event={"ID":"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd","Type":"ContainerStarted","Data":"6fa73b724f9b2e8dbaf6e2796e127457ea268b3d6d14a9124ba1cb32392fd869"}
Nov 03 16:02:22.702262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.702244    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4wvw" event={"ID":"66c8687d-b9da-46b2-85ce-52cd8f6ac1dd","Type":"ContainerStarted","Data":"488b22f8213698f5bb44d5248e29ad832c82cd80b1386361d68785ebad466b48"}
Nov 03 16:02:22.711505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:22.711461    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:23.345789 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.345702    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:23.345789 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.345758    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:23.345789 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.345782    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:23.346026 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:23.345852    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:02:23.346026 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:23.345877    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:23.346026 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:23.345936    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert podName:6ab65c33-a23d-48f3-ba16-212af80c62cb nodeName:}" failed. No retries permitted until 2026-11-03 16:02:27.345914178 +0000 UTC m=+8.859295933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8bw8n" (UID: "6ab65c33-a23d-48f3-ba16-212af80c62cb") : secret "networking-console-plugin-cert" not found
Nov 03 16:02:23.346026 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:23.345958    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls podName:4beaa779-098d-4007-88f8-25796f1758f0 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:27.345948538 +0000 UTC m=+8.859330292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcvq5" (UID: "4beaa779-098d-4007-88f8-25796f1758f0") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:02:23.348187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.348161    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a6ba52-20d8-40fe-9723-54418819a3ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7ql5t\" (UID: \"d7a6ba52-20d8-40fe-9723-54418819a3ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:23.478510 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.478470    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"
Nov 03 16:02:23.705626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.705593    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:23.707517 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.707493    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-859db4d5f7-xhncb"
Nov 03 16:02:23.709901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.709874    4964 generic.go:356] "Generic (PLEG): container finished" podID="bb4f4939-d15b-4b06-bc6e-27ca8b2039ec" containerID="2e6824e3e27dd9d188e0c6d8af180b09da37caebdeb1259041a092a7e8c88b77" exitCode=0
Nov 03 16:02:23.710046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.709948    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vrzhj" event={"ID":"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec","Type":"ContainerDied","Data":"2e6824e3e27dd9d188e0c6d8af180b09da37caebdeb1259041a092a7e8c88b77"}
Nov 03 16:02:23.710724 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.710708    4964 scope.go:122] "RemoveContainer" containerID="2e6824e3e27dd9d188e0c6d8af180b09da37caebdeb1259041a092a7e8c88b77"
Nov 03 16:02:23.752174 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:23.752144    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t"]
Nov 03 16:02:23.755820 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:23.755798    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a6ba52_20d8_40fe_9723_54418819a3ab.slice/crio-b72a9572d7e6622a62bed5fce19c36e13f695b4648da5beb1cb06df2656f7baa WatchSource:0}: Error finding container b72a9572d7e6622a62bed5fce19c36e13f695b4648da5beb1cb06df2656f7baa: Status 404 returned error can't find the container with id b72a9572d7e6622a62bed5fce19c36e13f695b4648da5beb1cb06df2656f7baa
Nov 03 16:02:24.448380 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.448346    4964 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:24.448845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.448829    4964 scope.go:122] "RemoveContainer" containerID="da0f5652214ce00301e16c11d816367d2b7d9a80ece12dad5c3d983f7f5593ed"
Nov 03 16:02:24.449059 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:24.449040    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4ftv2_openshift-console-operator(1b592ced-7c65-40a8-94d7-cad2c29fe081)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" podUID="1b592ced-7c65-40a8-94d7-cad2c29fe081"
Nov 03 16:02:24.717823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.717735    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vrzhj" event={"ID":"bb4f4939-d15b-4b06-bc6e-27ca8b2039ec","Type":"ContainerStarted","Data":"8c4c989e98830ac0d1358a34070194d88e96024fcfead83d8bb205f33317f2f1"}
Nov 03 16:02:24.720326 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.720294    4964 generic.go:356] "Generic (PLEG): container finished" podID="a2e92347-295e-4c8e-bef8-519fa58d4e87" containerID="ac9c321bf25bc75fa90a07216a4af7cbe966ab40daf0e5adfabf2d43fa45f07a" exitCode=0
Nov 03 16:02:24.720463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.720334    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q" event={"ID":"a2e92347-295e-4c8e-bef8-519fa58d4e87","Type":"ContainerDied","Data":"ac9c321bf25bc75fa90a07216a4af7cbe966ab40daf0e5adfabf2d43fa45f07a"}
Nov 03 16:02:24.720654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.720641    4964 scope.go:122] "RemoveContainer" containerID="ac9c321bf25bc75fa90a07216a4af7cbe966ab40daf0e5adfabf2d43fa45f07a"
Nov 03 16:02:24.722444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.722419    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t" event={"ID":"d7a6ba52-20d8-40fe-9723-54418819a3ab","Type":"ContainerStarted","Data":"b72a9572d7e6622a62bed5fce19c36e13f695b4648da5beb1cb06df2656f7baa"}
Nov 03 16:02:24.742925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.742898    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:24.743302 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:24.743282    4964 scope.go:122] "RemoveContainer" containerID="da0f5652214ce00301e16c11d816367d2b7d9a80ece12dad5c3d983f7f5593ed"
Nov 03 16:02:24.743579 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:24.743546    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4ftv2_openshift-console-operator(1b592ced-7c65-40a8-94d7-cad2c29fe081)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" podUID="1b592ced-7c65-40a8-94d7-cad2c29fe081"
Nov 03 16:02:25.218294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:25.218255    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:25.730799 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:25.730754    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f849q" event={"ID":"a2e92347-295e-4c8e-bef8-519fa58d4e87","Type":"ContainerStarted","Data":"071763a5f8e258c1c3043aea4dfd46421a835e9f2ec907b50732f987ad775b9c"}
Nov 03 16:02:25.732787 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:25.732753    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7ql5t" event={"ID":"d7a6ba52-20d8-40fe-9723-54418819a3ab","Type":"ContainerStarted","Data":"a34bb054c0e1009c4f7815c2272efc38f87719b4514a4fc1f98ee3c63732f2b3"}
Nov 03 16:02:26.242990 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:26.242959    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:26.243186 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:26.243173    4964 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:02:27.383039 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.382985    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:27.383482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.383097    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:27.385461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.385432    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4beaa779-098d-4007-88f8-25796f1758f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcvq5\" (UID: \"4beaa779-098d-4007-88f8-25796f1758f0\") " pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:27.385628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.385612    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab65c33-a23d-48f3-ba16-212af80c62cb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8bw8n\" (UID: \"6ab65c33-a23d-48f3-ba16-212af80c62cb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:27.391483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.391455    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bcvq5"
Nov 03 16:02:27.670098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.670066    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bcvq5"]
Nov 03 16:02:27.673200 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:27.673172    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beaa779_098d_4007_88f8_25796f1758f0.slice/crio-6619ea1f56854869c923cc9ab4df74e26d87ae4ee2a88c82330ecc8a1770d74d WatchSource:0}: Error finding container 6619ea1f56854869c923cc9ab4df74e26d87ae4ee2a88c82330ecc8a1770d74d: Status 404 returned error can't find the container with id 6619ea1f56854869c923cc9ab4df74e26d87ae4ee2a88c82330ecc8a1770d74d
Nov 03 16:02:27.679738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.679716    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"
Nov 03 16:02:27.746462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.744715    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcvq5" event={"ID":"4beaa779-098d-4007-88f8-25796f1758f0","Type":"ContainerStarted","Data":"6619ea1f56854869c923cc9ab4df74e26d87ae4ee2a88c82330ecc8a1770d74d"}
Nov 03 16:02:27.959778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:27.959689    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8bw8n"]
Nov 03 16:02:27.963316 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:27.963283    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab65c33_a23d_48f3_ba16_212af80c62cb.slice/crio-85fe23d1c8ed3b245f8f1ef585fced98f55107cc8beca28d54d58cfa3c8e86d3 WatchSource:0}: Error finding container 85fe23d1c8ed3b245f8f1ef585fced98f55107cc8beca28d54d58cfa3c8e86d3: Status 404 returned error can't find the container with id 85fe23d1c8ed3b245f8f1ef585fced98f55107cc8beca28d54d58cfa3c8e86d3
Nov 03 16:02:28.652805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:28.652769    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:28.750072 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:28.750036    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n" event={"ID":"6ab65c33-a23d-48f3-ba16-212af80c62cb","Type":"ContainerStarted","Data":"85fe23d1c8ed3b245f8f1ef585fced98f55107cc8beca28d54d58cfa3c8e86d3"}
Nov 03 16:02:28.752126 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:28.752091    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcvq5" event={"ID":"4beaa779-098d-4007-88f8-25796f1758f0","Type":"ContainerStarted","Data":"c9f2840ff748a9933c113617681c3f8f8718ab6662b139a411af61f5919a4974"}
Nov 03 16:02:28.752251 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:28.752159    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcvq5" event={"ID":"4beaa779-098d-4007-88f8-25796f1758f0","Type":"ContainerStarted","Data":"2a82fbe9b8bdeb914cceb01f7e4daa3d9af8164f422c9f3f9d91d423c12ab4ac"}
Nov 03 16:02:29.758889 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:29.758852    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8bw8n" event={"ID":"6ab65c33-a23d-48f3-ba16-212af80c62cb","Type":"ContainerStarted","Data":"04614f1e645a9d413164e0461d7bf5749de9012d06d1ab970aec159ca0049231"}
Nov 03 16:02:30.764684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:30.764638    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcvq5" event={"ID":"4beaa779-098d-4007-88f8-25796f1758f0","Type":"ContainerStarted","Data":"58246caae1755379cb64de9fde8119303fad195592b3ed2712cbf5568fc90b15"}
Nov 03 16:02:32.570136 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:02:32.570058    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:02:32.601800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:32.601768    4964 scope.go:122] "RemoveContainer" containerID="da0f5652214ce00301e16c11d816367d2b7d9a80ece12dad5c3d983f7f5593ed"
Nov 03 16:02:32.791944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:32.791915    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:02:32.792085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:32.792035    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2" event={"ID":"1b592ced-7c65-40a8-94d7-cad2c29fe081","Type":"ContainerStarted","Data":"3677c8cbd2ec1a7e6ec92ea95d21cb56f857c6f9d9492d3d46a67af347ac6777"}
Nov 03 16:02:32.792445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:32.792416    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:33.031839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.031810    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4ftv2"
Nov 03 16:02:33.308275 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.308183    4964 patch_prober.go:29] interesting pod/image-registry-7b94f5bc97-w9xjm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 16:02:33.308471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.308283    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm" podUID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:02:33.611857 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.609923    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-49czb-t8nxv"]
Nov 03 16:02:33.615894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.615849    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.629397 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.629302    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-49czb-t8nxv"]
Nov 03 16:02:33.657995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.656547    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"]
Nov 03 16:02:33.660204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.660174    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:33.666438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.665317    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"]
Nov 03 16:02:33.673940 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.673911    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wgpsc\""
Nov 03 16:02:33.674206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.674099    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 16:02:33.682807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.682781    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d76ff499b-x7d7l"]
Nov 03 16:02:33.724921 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.724894    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b4wvw"
Nov 03 16:02:33.750050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.750009    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rj9\" (UniqueName: \"kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.750244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.750059    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.750244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.750093    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.750244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.750142    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c2be6f7-3fae-4cb9-9615-3a3d15140cec-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7dmkl\" (UID: \"3c2be6f7-3fae-4cb9-9615-3a3d15140cec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:33.750244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.750185    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.750480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.750451    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"]
Nov 03 16:02:33.754226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.754197    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-xrkn4"]
Nov 03 16:02:33.754380 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.754314    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"
Nov 03 16:02:33.758040 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.758011    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.762601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.762571    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-556f6f5cbf-277c2"]
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.766722    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.771544    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"]
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.775091    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.775237    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.775345    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zcs66\""
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.775458    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.775543    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 16:02:33.775932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.775863    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qd7db\""
Nov 03 16:02:33.779062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.779038    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"]
Nov 03 16:02:33.779376 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.779323    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"
Nov 03 16:02:33.779846 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.779821    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xrkn4"]
Nov 03 16:02:33.785016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.784031    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556f6f5cbf-277c2"]
Nov 03 16:02:33.806730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.805625    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"]
Nov 03 16:02:33.857004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.856957    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab643762-ad17-432d-9190-b1c6fd2f6a34-installation-pull-secrets\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857177 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857017    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c2be6f7-3fae-4cb9-9615-3a3d15140cec-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7dmkl\" (UID: \"3c2be6f7-3fae-4cb9-9615-3a3d15140cec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:33.857177 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857089    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.857177 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857136    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-bound-sa-token\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857351 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857190    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9z8\" (UniqueName: \"kubernetes.io/projected/882a2834-bc7e-4500-acc2-33261144209e-kube-api-access-gm9z8\") pod \"migrator-66df5d496b-24rtv\" (UID: \"882a2834-bc7e-4500-acc2-33261144209e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"
Nov 03 16:02:33.857351 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857317    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab643762-ad17-432d-9190-b1c6fd2f6a34-ca-trust-extracted\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857447    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rj9\" (UniqueName: \"kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.857522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857500    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64dd3f55-3e92-4fb9-a122-694296b32596-tmp\") pod \"downloads-6dd789c8b7-xrkn4\" (UID: \"64dd3f55-3e92-4fb9-a122-694296b32596\") " pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.857570 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857527    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgr5\" (UniqueName: \"kubernetes.io/projected/64dd3f55-3e92-4fb9-a122-694296b32596-kube-api-access-xkgr5\") pod \"downloads-6dd789c8b7-xrkn4\" (UID: \"64dd3f55-3e92-4fb9-a122-694296b32596\") " pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.857570 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857562    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8np\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-kube-api-access-vq8np\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857588    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.857665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857632    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.857665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857654    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx5pr\" (UniqueName: \"kubernetes.io/projected/a6c06fb9-f4fd-4688-a449-ba44389d9785-kube-api-access-lx5pr\") pod \"migrator-66df5d496b-9rtv8\" (UID: \"a6c06fb9-f4fd-4688-a449-ba44389d9785\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"
Nov 03 16:02:33.857806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857676    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-registry-tls\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857704    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab643762-ad17-432d-9190-b1c6fd2f6a34-registry-certificates\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857728    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab643762-ad17-432d-9190-b1c6fd2f6a34-trusted-ca\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.857806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.857754    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab643762-ad17-432d-9190-b1c6fd2f6a34-image-registry-private-configuration\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.858203 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.858173    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.858360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.858327    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.858620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.858592    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.859951 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.859925    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c2be6f7-3fae-4cb9-9615-3a3d15140cec-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7dmkl\" (UID: \"3c2be6f7-3fae-4cb9-9615-3a3d15140cec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:33.874988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.874915    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rj9\" (UniqueName: \"kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9\") pod \"periodic-gathering-49czb-t8nxv\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") " pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.933131 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.933100    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:02:33.958221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958186    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab643762-ad17-432d-9190-b1c6fd2f6a34-installation-pull-secrets\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958262    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-bound-sa-token\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958315    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9z8\" (UniqueName: \"kubernetes.io/projected/882a2834-bc7e-4500-acc2-33261144209e-kube-api-access-gm9z8\") pod \"migrator-66df5d496b-24rtv\" (UID: \"882a2834-bc7e-4500-acc2-33261144209e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"
Nov 03 16:02:33.958420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958365    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab643762-ad17-432d-9190-b1c6fd2f6a34-ca-trust-extracted\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958415    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64dd3f55-3e92-4fb9-a122-694296b32596-tmp\") pod \"downloads-6dd789c8b7-xrkn4\" (UID: \"64dd3f55-3e92-4fb9-a122-694296b32596\") " pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958437    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgr5\" (UniqueName: \"kubernetes.io/projected/64dd3f55-3e92-4fb9-a122-694296b32596-kube-api-access-xkgr5\") pod \"downloads-6dd789c8b7-xrkn4\" (UID: \"64dd3f55-3e92-4fb9-a122-694296b32596\") " pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958453    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8np\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-kube-api-access-vq8np\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958478    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx5pr\" (UniqueName: \"kubernetes.io/projected/a6c06fb9-f4fd-4688-a449-ba44389d9785-kube-api-access-lx5pr\") pod \"migrator-66df5d496b-9rtv8\" (UID: \"a6c06fb9-f4fd-4688-a449-ba44389d9785\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958492    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-registry-tls\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958512    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab643762-ad17-432d-9190-b1c6fd2f6a34-registry-certificates\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958529    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab643762-ad17-432d-9190-b1c6fd2f6a34-trusted-ca\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.958646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.958561    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab643762-ad17-432d-9190-b1c6fd2f6a34-image-registry-private-configuration\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.959587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.959168    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab643762-ad17-432d-9190-b1c6fd2f6a34-ca-trust-extracted\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.959587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.959571    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64dd3f55-3e92-4fb9-a122-694296b32596-tmp\") pod \"downloads-6dd789c8b7-xrkn4\" (UID: \"64dd3f55-3e92-4fb9-a122-694296b32596\") " pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.959779 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.959688    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab643762-ad17-432d-9190-b1c6fd2f6a34-registry-certificates\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.959846 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.959802    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab643762-ad17-432d-9190-b1c6fd2f6a34-trusted-ca\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.961568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.961547    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab643762-ad17-432d-9190-b1c6fd2f6a34-image-registry-private-configuration\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.961786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.961768    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-registry-tls\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.961914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.961894    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab643762-ad17-432d-9190-b1c6fd2f6a34-installation-pull-secrets\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.978605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.978535    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9z8\" (UniqueName: \"kubernetes.io/projected/882a2834-bc7e-4500-acc2-33261144209e-kube-api-access-gm9z8\") pod \"migrator-66df5d496b-24rtv\" (UID: \"882a2834-bc7e-4500-acc2-33261144209e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"
Nov 03 16:02:33.978777 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.978609    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-bound-sa-token\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.978777 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.978673    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx5pr\" (UniqueName: \"kubernetes.io/projected/a6c06fb9-f4fd-4688-a449-ba44389d9785-kube-api-access-lx5pr\") pod \"migrator-66df5d496b-9rtv8\" (UID: \"a6c06fb9-f4fd-4688-a449-ba44389d9785\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"
Nov 03 16:02:33.981691 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.981653    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8np\" (UniqueName: \"kubernetes.io/projected/ab643762-ad17-432d-9190-b1c6fd2f6a34-kube-api-access-vq8np\") pod \"image-registry-556f6f5cbf-277c2\" (UID: \"ab643762-ad17-432d-9190-b1c6fd2f6a34\") " pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:33.981940 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.981920    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgr5\" (UniqueName: \"kubernetes.io/projected/64dd3f55-3e92-4fb9-a122-694296b32596-kube-api-access-xkgr5\") pod \"downloads-6dd789c8b7-xrkn4\" (UID: \"64dd3f55-3e92-4fb9-a122-694296b32596\") " pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:33.985810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:33.985787    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:34.098565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.098526    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"
Nov 03 16:02:34.107930 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.107903    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:34.118221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.118188    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:34.125593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.125352    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"
Nov 03 16:02:34.227254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.227115    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-49czb-t8nxv"]
Nov 03 16:02:34.230369 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:34.230333    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb7e596_9c64_4bcc_8841_45d1fd5e0485.slice/crio-d3429c4dbfcd807133900ea719abce77c3d7b4d1ab5b15a7eb91ebdb6ebc078e WatchSource:0}: Error finding container d3429c4dbfcd807133900ea719abce77c3d7b4d1ab5b15a7eb91ebdb6ebc078e: Status 404 returned error can't find the container with id d3429c4dbfcd807133900ea719abce77c3d7b4d1ab5b15a7eb91ebdb6ebc078e
Nov 03 16:02:34.273886 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.273774    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"]
Nov 03 16:02:34.276262 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:34.276209    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2be6f7_3fae_4cb9_9615_3a3d15140cec.slice/crio-b53460f19b9f2eb27e182e65b412b9507f6978e8405a31b19236e0610a9e53cd WatchSource:0}: Error finding container b53460f19b9f2eb27e182e65b412b9507f6978e8405a31b19236e0610a9e53cd: Status 404 returned error can't find the container with id b53460f19b9f2eb27e182e65b412b9507f6978e8405a31b19236e0610a9e53cd
Nov 03 16:02:34.400086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.400050    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8"]
Nov 03 16:02:34.402499 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:34.402458    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c06fb9_f4fd_4688_a449_ba44389d9785.slice/crio-5df0945a25d2fb93dd8c421cf52739edd450e09a7211d7b5792380dd6de553d2 WatchSource:0}: Error finding container 5df0945a25d2fb93dd8c421cf52739edd450e09a7211d7b5792380dd6de553d2: Status 404 returned error can't find the container with id 5df0945a25d2fb93dd8c421cf52739edd450e09a7211d7b5792380dd6de553d2
Nov 03 16:02:34.411759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.411732    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xrkn4"]
Nov 03 16:02:34.414544 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:34.414515    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dd3f55_3e92_4fb9_a122_694296b32596.slice/crio-02f4514b949a8be76ebf2346c7aedc729ae0ba5c64a04593e21130c2cf42bd98 WatchSource:0}: Error finding container 02f4514b949a8be76ebf2346c7aedc729ae0ba5c64a04593e21130c2cf42bd98: Status 404 returned error can't find the container with id 02f4514b949a8be76ebf2346c7aedc729ae0ba5c64a04593e21130c2cf42bd98
Nov 03 16:02:34.422562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.422534    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv"]
Nov 03 16:02:34.425729 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:34.425703    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882a2834_bc7e_4500_acc2_33261144209e.slice/crio-bba8cb437f392b0ea84adbca7172a1edb1b2775bd8b752c26d867bb88da48dd6 WatchSource:0}: Error finding container bba8cb437f392b0ea84adbca7172a1edb1b2775bd8b752c26d867bb88da48dd6: Status 404 returned error can't find the container with id bba8cb437f392b0ea84adbca7172a1edb1b2775bd8b752c26d867bb88da48dd6
Nov 03 16:02:34.439566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.439536    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556f6f5cbf-277c2"]
Nov 03 16:02:34.444438 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:34.444373    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab643762_ad17_432d_9190_b1c6fd2f6a34.slice/crio-9e0765fc046b8b5d5401b2ecfa78fe458ad3e86fddc2a6d3b441678c41681892 WatchSource:0}: Error finding container 9e0765fc046b8b5d5401b2ecfa78fe458ad3e86fddc2a6d3b441678c41681892: Status 404 returned error can't find the container with id 9e0765fc046b8b5d5401b2ecfa78fe458ad3e86fddc2a6d3b441678c41681892
Nov 03 16:02:34.820292 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.820259    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2" event={"ID":"ab643762-ad17-432d-9190-b1c6fd2f6a34","Type":"ContainerStarted","Data":"adbce8bb5eaad34fd7df09bb2f70d759f6b6803a98d2960b50513708bb8078b7"}
Nov 03 16:02:34.820839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.820317    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2" event={"ID":"ab643762-ad17-432d-9190-b1c6fd2f6a34","Type":"ContainerStarted","Data":"9e0765fc046b8b5d5401b2ecfa78fe458ad3e86fddc2a6d3b441678c41681892"}
Nov 03 16:02:34.820839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.820374    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:34.822213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.822185    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8" event={"ID":"a6c06fb9-f4fd-4688-a449-ba44389d9785","Type":"ContainerStarted","Data":"5df0945a25d2fb93dd8c421cf52739edd450e09a7211d7b5792380dd6de553d2"}
Nov 03 16:02:34.824012 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.823982    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-49czb-t8nxv" event={"ID":"4cb7e596-9c64-4bcc-8841-45d1fd5e0485","Type":"ContainerStarted","Data":"085a2324a2130420251d229ef543af194bc127886a1689b679f17cb2a9f5d150"}
Nov 03 16:02:34.824127 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.824027    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-49czb-t8nxv" event={"ID":"4cb7e596-9c64-4bcc-8841-45d1fd5e0485","Type":"ContainerStarted","Data":"d3429c4dbfcd807133900ea719abce77c3d7b4d1ab5b15a7eb91ebdb6ebc078e"}
Nov 03 16:02:34.825243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.825217    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv" event={"ID":"882a2834-bc7e-4500-acc2-33261144209e","Type":"ContainerStarted","Data":"bba8cb437f392b0ea84adbca7172a1edb1b2775bd8b752c26d867bb88da48dd6"}
Nov 03 16:02:34.826361 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.826339    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xrkn4" event={"ID":"64dd3f55-3e92-4fb9-a122-694296b32596","Type":"ContainerStarted","Data":"02f4514b949a8be76ebf2346c7aedc729ae0ba5c64a04593e21130c2cf42bd98"}
Nov 03 16:02:34.827973 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.827948    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl" event={"ID":"3c2be6f7-3fae-4cb9-9615-3a3d15140cec","Type":"ContainerStarted","Data":"b53460f19b9f2eb27e182e65b412b9507f6978e8405a31b19236e0610a9e53cd"}
Nov 03 16:02:34.849265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.849152    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2" podStartSLOduration=1.849136685 podStartE2EDuration="1.849136685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:34.848833724 +0000 UTC m=+16.362215499" watchObservedRunningTime="2026-11-03 16:02:34.849136685 +0000 UTC m=+16.362518489"
Nov 03 16:02:34.871114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:34.871047    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-49czb-t8nxv" podStartSLOduration=1.871027267 podStartE2EDuration="1.871027267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:34.867771376 +0000 UTC m=+16.381153146" watchObservedRunningTime="2026-11-03 16:02:34.871027267 +0000 UTC m=+16.384409046"
Nov 03 16:02:35.884099 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:35.884060    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4wvw_66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/dns/0.log"
Nov 03 16:02:36.089250 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.089220    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4wvw_66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/kube-rbac-proxy/0.log"
Nov 03 16:02:36.356767 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.356688    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r59q4"
Nov 03 16:02:36.484118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.484083    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mlccd_6a30fac5-ac40-4179-9628-fa3203ddb77b/dns-node-resolver/0.log"
Nov 03 16:02:36.841819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.841784    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv" event={"ID":"882a2834-bc7e-4500-acc2-33261144209e","Type":"ContainerStarted","Data":"259d73dfc236c40cbbba9eed693364d6dbef3327aa6368aa7aa5f28c82c9a352"}
Nov 03 16:02:36.843493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.843456    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl" event={"ID":"3c2be6f7-3fae-4cb9-9615-3a3d15140cec","Type":"ContainerStarted","Data":"b81a654d71325a60b9c8f6450f263ad95636bdd5d04d8502670e0515a78b123c"}
Nov 03 16:02:36.843636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.843518    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:36.844984 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.844954    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8" event={"ID":"a6c06fb9-f4fd-4688-a449-ba44389d9785","Type":"ContainerStarted","Data":"c9e3c74be3a755df4c213a5bc9e59482e1f1240fbfac3382ef7b80eebf70d71d"}
Nov 03 16:02:36.849657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.849630    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl"
Nov 03 16:02:36.872982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.872917    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rtv" podStartSLOduration=2.344353617 podStartE2EDuration="3.872897196s" totalImagesPullingTime="1.528543579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:36.871464358 +0000 UTC m=+18.384846137" watchObservedRunningTime="2026-11-03 16:02:36.872897196 +0000 UTC m=+18.386278972"
Nov 03 16:02:36.888439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.888409    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-556f6f5cbf-277c2_ab643762-ad17-432d-9190-b1c6fd2f6a34/registry/0.log"
Nov 03 16:02:36.891709 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.891654    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dmkl" podStartSLOduration=2.381145444 podStartE2EDuration="3.891637406s" totalImagesPullingTime="1.510491962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:36.891450823 +0000 UTC m=+18.404832592" watchObservedRunningTime="2026-11-03 16:02:36.891637406 +0000 UTC m=+18.405019184"
Nov 03 16:02:36.912362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:36.912312    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9rtv8" podStartSLOduration=2.357814165 podStartE2EDuration="3.912294292s" totalImagesPullingTime="1.554480127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:36.911749971 +0000 UTC m=+18.425131748" watchObservedRunningTime="2026-11-03 16:02:36.912294292 +0000 UTC m=+18.425676069"
Nov 03 16:02:37.083681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.083655    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d76ff499b-x7d7l_df53f549-e19a-4304-8387-4afa8528b975/registry/0.log"
Nov 03 16:02:37.206070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.206031    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"]
Nov 03 16:02:37.209651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.209629    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.218034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.218004    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 16:02:37.218158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.218004    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 16:02:37.219034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.219016    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7jqp4\""
Nov 03 16:02:37.219137 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.219122    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 16:02:37.223630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.223554    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"]
Nov 03 16:02:37.286351 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.286312    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b94f5bc97-w9xjm_17eca28e-8ea1-4193-92c1-32dd16b78e0d/registry/0.log"
Nov 03 16:02:37.291963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.291926    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f35af85a-5764-4e7d-a673-3e7fb1883d0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.292132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.291980    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f35af85a-5764-4e7d-a673-3e7fb1883d0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.292132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.292067    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74t8m\" (UniqueName: \"kubernetes.io/projected/f35af85a-5764-4e7d-a673-3e7fb1883d0d-kube-api-access-74t8m\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.292199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.292126    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35af85a-5764-4e7d-a673-3e7fb1883d0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.393215 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.393176    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35af85a-5764-4e7d-a673-3e7fb1883d0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.393366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.393231    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f35af85a-5764-4e7d-a673-3e7fb1883d0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.393366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.393265    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f35af85a-5764-4e7d-a673-3e7fb1883d0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.393366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.393290    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74t8m\" (UniqueName: \"kubernetes.io/projected/f35af85a-5764-4e7d-a673-3e7fb1883d0d-kube-api-access-74t8m\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.393929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.393902    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35af85a-5764-4e7d-a673-3e7fb1883d0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.395760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.395733    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f35af85a-5764-4e7d-a673-3e7fb1883d0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.395841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.395739    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f35af85a-5764-4e7d-a673-3e7fb1883d0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.406700 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.406658    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74t8m\" (UniqueName: \"kubernetes.io/projected/f35af85a-5764-4e7d-a673-3e7fb1883d0d-kube-api-access-74t8m\") pod \"prometheus-operator-9cd89b74c-bzxr6\" (UID: \"f35af85a-5764-4e7d-a673-3e7fb1883d0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.483697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.483610    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5wvsc_66a8e707-3e0b-42f2-988f-517d834a869c/node-ca/0.log"
Nov 03 16:02:37.525652 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.525592    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"
Nov 03 16:02:37.819080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.819002    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6"]
Nov 03 16:02:37.822237 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:37.822213    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35af85a_5764_4e7d_a673_3e7fb1883d0d.slice/crio-db2101ad5525aa8b5227accffc1bec07eb7c726cc90d2954cb00f8a39e55fa45 WatchSource:0}: Error finding container db2101ad5525aa8b5227accffc1bec07eb7c726cc90d2954cb00f8a39e55fa45: Status 404 returned error can't find the container with id db2101ad5525aa8b5227accffc1bec07eb7c726cc90d2954cb00f8a39e55fa45
Nov 03 16:02:37.851152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:37.851109    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6" event={"ID":"f35af85a-5764-4e7d-a673-3e7fb1883d0d","Type":"ContainerStarted","Data":"db2101ad5525aa8b5227accffc1bec07eb7c726cc90d2954cb00f8a39e55fa45"}
Nov 03 16:02:38.084426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:38.084328    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-859db4d5f7-xhncb_2351ca8a-ffba-4733-8413-2ae0b585b489/router/0.log"
Nov 03 16:02:38.283463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:38.283434    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8qq6k_e3dbcf1c-4684-46e8-b806-1df38410a5c9/serve-healthcheck-canary/0.log"
Nov 03 16:02:38.484275 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:38.484244    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-24rtv_882a2834-bc7e-4500-acc2-33261144209e/migrator/0.log"
Nov 03 16:02:38.684258 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:38.684194    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-9rtv8_a6c06fb9-f4fd-4688-a449-ba44389d9785/migrator/0.log"
Nov 03 16:02:38.886198 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:38.886130    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-f849q_a2e92347-295e-4c8e-bef8-519fa58d4e87/kube-storage-version-migrator-operator/0.log"
Nov 03 16:02:39.092450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:39.092420    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-f849q_a2e92347-295e-4c8e-bef8-519fa58d4e87/kube-storage-version-migrator-operator/1.log"
Nov 03 16:02:40.869141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:40.869096    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6" event={"ID":"f35af85a-5764-4e7d-a673-3e7fb1883d0d","Type":"ContainerStarted","Data":"4ff73ac860c4d2cdf0fa8febf42f3945ce052e57b5a9880c6437e391129fcfea"}
Nov 03 16:02:40.869606 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:40.869167    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6" event={"ID":"f35af85a-5764-4e7d-a673-3e7fb1883d0d","Type":"ContainerStarted","Data":"5a932db908b1b32d680f388acde44d87309c164264e85beef20a871a4f372722"}
Nov 03 16:02:40.891476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:40.891417    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bzxr6" podStartSLOduration=1.858775052 podStartE2EDuration="3.891401327s" totalImagesPullingTime="2.032626275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:40.890544775 +0000 UTC m=+22.403926552" watchObservedRunningTime="2026-11-03 16:02:40.891401327 +0000 UTC m=+22.404783097"
Nov 03 16:02:42.622452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.622414    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t4w2k"]
Nov 03 16:02:42.655975 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.655944    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.677180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.677143    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrtx4\""
Nov 03 16:02:42.677682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.677641    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 16:02:42.677818 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.677754    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 16:02:42.708498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.708436    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:02:42.724112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.724077    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 16:02:42.845790 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.845755    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-accelerators-collector-config\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.845996 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.845815    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-root\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846098    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-wtmp\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846203 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846161    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-textfile\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846212    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rvw\" (UniqueName: \"kubernetes.io/projected/70b0c8dc-d244-46ea-a229-389133d58c51-kube-api-access-c6rvw\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846336 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846319    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-tls\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846369    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846467    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b0c8dc-d244-46ea-a229-389133d58c51-metrics-client-ca\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.846536 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.846525    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-sys\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947597 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947554    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rvw\" (UniqueName: \"kubernetes.io/projected/70b0c8dc-d244-46ea-a229-389133d58c51-kube-api-access-c6rvw\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947715    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-tls\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947745    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947782    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b0c8dc-d244-46ea-a229-389133d58c51-metrics-client-ca\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947819    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-sys\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947841    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-accelerators-collector-config\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947861    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-root\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947894    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-wtmp\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947921    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-textfile\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.947993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.947936    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-sys\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.948272 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.948045    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-root\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.948272 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.948129    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-wtmp\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.948528 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.948495    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-textfile\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.948878 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.948848    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-accelerators-collector-config\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.949027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.948875    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b0c8dc-d244-46ea-a229-389133d58c51-metrics-client-ca\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.950421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.950378    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-tls\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.950562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.950528    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b0c8dc-d244-46ea-a229-389133d58c51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:42.969465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:42.969428    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rvw\" (UniqueName: \"kubernetes.io/projected/70b0c8dc-d244-46ea-a229-389133d58c51-kube-api-access-c6rvw\") pod \"node-exporter-t4w2k\" (UID: \"70b0c8dc-d244-46ea-a229-389133d58c51\") " pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:43.034472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:43.034439    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t4w2k"
Nov 03 16:02:43.084154 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:02:43.084113    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b0c8dc_d244_46ea_a229_389133d58c51.slice/crio-04afd4c4ae15aa09ba3f2922021abadd86a11d838c3082df822786764578395a WatchSource:0}: Error finding container 04afd4c4ae15aa09ba3f2922021abadd86a11d838c3082df822786764578395a: Status 404 returned error can't find the container with id 04afd4c4ae15aa09ba3f2922021abadd86a11d838c3082df822786764578395a
Nov 03 16:02:43.693692 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:43.693652    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:43.886159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:43.886121    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4w2k" event={"ID":"70b0c8dc-d244-46ea-a229-389133d58c51","Type":"ContainerStarted","Data":"04afd4c4ae15aa09ba3f2922021abadd86a11d838c3082df822786764578395a"}
Nov 03 16:02:44.893149 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:44.893071    4964 generic.go:356] "Generic (PLEG): container finished" podID="70b0c8dc-d244-46ea-a229-389133d58c51" containerID="589fe4a1bd36eb9b59d9720314d7e3a465438ff44c7ca4bd0856433ecd6853ba" exitCode=0
Nov 03 16:02:44.893149 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:44.893109    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4w2k" event={"ID":"70b0c8dc-d244-46ea-a229-389133d58c51","Type":"ContainerDied","Data":"589fe4a1bd36eb9b59d9720314d7e3a465438ff44c7ca4bd0856433ecd6853ba"}
Nov 03 16:02:44.943904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:44.943873    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h45wd"
Nov 03 16:02:45.900332 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:45.900295    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4w2k" event={"ID":"70b0c8dc-d244-46ea-a229-389133d58c51","Type":"ContainerStarted","Data":"7760e8b74d923f135051ff2b8ec50d77a299af998c015abcfa894405eb0c266a"}
Nov 03 16:02:45.900775 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:45.900355    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4w2k" event={"ID":"70b0c8dc-d244-46ea-a229-389133d58c51","Type":"ContainerStarted","Data":"8c718117205dce0d6298ba424dc9371dcfb107625421ec009418b7ff9dbb3864"}
Nov 03 16:02:45.923938 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:45.923862    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t4w2k" podStartSLOduration=2.302830465 podStartE2EDuration="3.923842518s" totalImagesPullingTime="1.521605549s" totalInitContainerRuntime="99.406504ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:45.923379118 +0000 UTC m=+27.436760884" watchObservedRunningTime="2026-11-03 16:02:45.923842518 +0000 UTC m=+27.437224297"
Nov 03 16:02:51.417932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:51.417890    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kdkkw"
Nov 03 16:02:55.842918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:55.842878    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-556f6f5cbf-277c2"
Nov 03 16:02:55.909704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:55.909661    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b94f5bc97-w9xjm"]
Nov 03 16:02:55.978453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:55.978413    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xrkn4" event={"ID":"64dd3f55-3e92-4fb9-a122-694296b32596","Type":"ContainerStarted","Data":"13f43d792de8d48bc1b7bfa052c66293a78cccb200953fd2d8fac9ff27345dd1"}
Nov 03 16:02:55.981527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:55.981304    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:55.995709 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:55.995492    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-xrkn4"
Nov 03 16:02:56.002665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:56.002080    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-xrkn4" podStartSLOduration=2.494156992 podStartE2EDuration="23.002058108s" totalImagesPullingTime="20.507901116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:56.000382472 +0000 UTC m=+37.513764274" watchObservedRunningTime="2026-11-03 16:02:56.002058108 +0000 UTC m=+37.515439884"
Nov 03 16:02:58.709154 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:58.709105    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l" podUID="df53f549-e19a-4304-8387-4afa8528b975" containerName="registry" containerID="cri-o://750d90006b95b446559274619a02c24158258f482ffa0f97a2f697ff1aac1b11" gracePeriod=30
Nov 03 16:02:58.996651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:58.996595    4964 generic.go:356] "Generic (PLEG): container finished" podID="df53f549-e19a-4304-8387-4afa8528b975" containerID="750d90006b95b446559274619a02c24158258f482ffa0f97a2f697ff1aac1b11" exitCode=0
Nov 03 16:02:58.996856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:58.996667    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l" event={"ID":"df53f549-e19a-4304-8387-4afa8528b975","Type":"ContainerDied","Data":"750d90006b95b446559274619a02c24158258f482ffa0f97a2f697ff1aac1b11"}
Nov 03 16:02:58.996856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:58.996759    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l" event={"ID":"df53f549-e19a-4304-8387-4afa8528b975","Type":"ContainerDied","Data":"7b7a2b63c99569b33fec7241c9d3cbfc89dab31bc8413cdc0a924e6e79401b06"}
Nov 03 16:02:58.996856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:58.996788    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b7a2b63c99569b33fec7241c9d3cbfc89dab31bc8413cdc0a924e6e79401b06"
Nov 03 16:02:59.025729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.025694    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:02:59.118112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118075    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118123    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118157    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118213    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118249    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118346    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118381    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118448    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") pod \"df53f549-e19a-4304-8387-4afa8528b975\" (UID: \"df53f549-e19a-4304-8387-4afa8528b975\") "
Nov 03 16:02:59.118725 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118592    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:02:59.118779 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118739    4964 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-registry-certificates\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.119447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.118959    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:02:59.122491 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.122453    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:02:59.127526 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.127461    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:02:59.129605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.127965    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:02:59.129605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.128098    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:02:59.129605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.128132    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "kube-api-access-2z9tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:02:59.129924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.129805    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token" pod "df53f549-e19a-4304-8387-4afa8528b975" (UID: "df53f549-e19a-4304-8387-4afa8528b975"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:02:59.219662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219619    4964 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-bound-sa-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.219662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219660    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df53f549-e19a-4304-8387-4afa8528b975-trusted-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.219890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219676    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9tc\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-kube-api-access-2z9tc\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.219890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219691    4964 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-installation-pull-secrets\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.219890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219704    4964 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df53f549-e19a-4304-8387-4afa8528b975-registry-tls\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.219890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219716    4964 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df53f549-e19a-4304-8387-4afa8528b975-ca-trust-extracted\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:02:59.219890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:02:59.219732    4964 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df53f549-e19a-4304-8387-4afa8528b975-image-registry-private-configuration\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:00.002114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:00.002077    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d76ff499b-x7d7l"
Nov 03 16:03:00.026926 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:00.026888    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d76ff499b-x7d7l"]
Nov 03 16:03:00.033431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:00.033398    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d76ff499b-x7d7l"]
Nov 03 16:03:00.608941 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:00.608883    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df53f549-e19a-4304-8387-4afa8528b975" path="/var/lib/kubelet/pods/df53f549-e19a-4304-8387-4afa8528b975/volumes"
Nov 03 16:03:02.573219 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:03:02.573173    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:03:05.854827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:05.854792    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-b4vvv-lcf5m"]
Nov 03 16:03:05.855453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:05.855005    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m" podUID="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" containerName="insights-gathering" containerID="cri-o://9d84057fe6988cea327584b7c0daa252502b8c1cc8f1987fb3ca195db75eb9b0" gracePeriod=30
Nov 03 16:03:06.036818 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.036784    4964 generic.go:356] "Generic (PLEG): container finished" podID="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" containerID="9d84057fe6988cea327584b7c0daa252502b8c1cc8f1987fb3ca195db75eb9b0" exitCode=2
Nov 03 16:03:06.036981 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.036856    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m" event={"ID":"0a088fef-29ab-4b9e-a90d-f6560aa4ee43","Type":"ContainerDied","Data":"9d84057fe6988cea327584b7c0daa252502b8c1cc8f1987fb3ca195db75eb9b0"}
Nov 03 16:03:06.201349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.201311    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:03:06.288910 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.288871    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") pod \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") "
Nov 03 16:03:06.289108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.288944    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") pod \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") "
Nov 03 16:03:06.289108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.289004    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") pod \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") "
Nov 03 16:03:06.289108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.289026    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") pod \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\" (UID: \"0a088fef-29ab-4b9e-a90d-f6560aa4ee43\") "
Nov 03 16:03:06.289314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.289278    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle" pod "0a088fef-29ab-4b9e-a90d-f6560aa4ee43" (UID: "0a088fef-29ab-4b9e-a90d-f6560aa4ee43"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:06.289362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.289311    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle" pod "0a088fef-29ab-4b9e-a90d-f6560aa4ee43" (UID: "0a088fef-29ab-4b9e-a90d-f6560aa4ee43"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:06.289452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.289435    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path" pod "0a088fef-29ab-4b9e-a90d-f6560aa4ee43" (UID: "0a088fef-29ab-4b9e-a90d-f6560aa4ee43"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:06.291249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.291222    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk" pod "0a088fef-29ab-4b9e-a90d-f6560aa4ee43" (UID: "0a088fef-29ab-4b9e-a90d-f6560aa4ee43"). InnerVolumeSpecName "kube-api-access-7pqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:06.390197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.390160    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pqfk\" (UniqueName: \"kubernetes.io/projected/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-kube-api-access-7pqfk\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:06.390197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.390193    4964 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-archives-path\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:06.390197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.390203    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-trusted-ca-bundle\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:06.390491 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:06.390213    4964 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a088fef-29ab-4b9e-a90d-f6560aa4ee43-service-ca-bundle\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:07.042160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:07.042121    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m" event={"ID":"0a088fef-29ab-4b9e-a90d-f6560aa4ee43","Type":"ContainerDied","Data":"94f497c6f15eab0f8fb2a755d014d7c0123d952628d974affc2e54aff2c129c9"}
Nov 03 16:03:07.042160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:07.042159    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b4vvv-lcf5m"
Nov 03 16:03:07.042748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:07.042183    4964 scope.go:122] "RemoveContainer" containerID="9d84057fe6988cea327584b7c0daa252502b8c1cc8f1987fb3ca195db75eb9b0"
Nov 03 16:03:07.060582 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:07.060549    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-b4vvv-lcf5m"]
Nov 03 16:03:07.065605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:07.065576    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-b4vvv-lcf5m"]
Nov 03 16:03:08.606531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:08.606488    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" path="/var/lib/kubelet/pods/0a088fef-29ab-4b9e-a90d-f6560aa4ee43/volumes"
Nov 03 16:03:20.957665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:20.957594    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm" podUID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" containerName="registry" containerID="cri-o://2923eeb96076e7d1b51c7bd922fa97d5a8c6bcbe9e8f187ca4a189d488e9df3c" gracePeriod=30
Nov 03 16:03:21.105369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.105337    4964 generic.go:356] "Generic (PLEG): container finished" podID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" containerID="2923eeb96076e7d1b51c7bd922fa97d5a8c6bcbe9e8f187ca4a189d488e9df3c" exitCode=0
Nov 03 16:03:21.105572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.105420    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm" event={"ID":"17eca28e-8ea1-4193-92c1-32dd16b78e0d","Type":"ContainerDied","Data":"2923eeb96076e7d1b51c7bd922fa97d5a8c6bcbe9e8f187ca4a189d488e9df3c"}
Nov 03 16:03:21.224110 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.224032    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:03:21.323004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.322960    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323016    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323064    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323112    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323144    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323433 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323207    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323433 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323234    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323433 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323313    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") pod \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\" (UID: \"17eca28e-8ea1-4193-92c1-32dd16b78e0d\") "
Nov 03 16:03:21.323579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323534    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:21.324002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.323950    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:21.325686 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.325648    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:21.325908 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.325870    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "kube-api-access-5ftxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:21.326015 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.325987    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:21.326069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.326049    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:21.326106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.326070    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:21.332457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.332380    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted" pod "17eca28e-8ea1-4193-92c1-32dd16b78e0d" (UID: "17eca28e-8ea1-4193-92c1-32dd16b78e0d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:21.424159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424125    4964 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-image-registry-private-configuration\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424155    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-trusted-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424166    4964 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17eca28e-8ea1-4193-92c1-32dd16b78e0d-ca-trust-extracted\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424175    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ftxg\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-kube-api-access-5ftxg\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424186    4964 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-bound-sa-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424194    4964 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-certificates\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424203    4964 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17eca28e-8ea1-4193-92c1-32dd16b78e0d-registry-tls\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:21.424381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:21.424211    4964 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17eca28e-8ea1-4193-92c1-32dd16b78e0d-installation-pull-secrets\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:03:22.111357 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:22.111330    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm"
Nov 03 16:03:22.111835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:22.111323    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b94f5bc97-w9xjm" event={"ID":"17eca28e-8ea1-4193-92c1-32dd16b78e0d","Type":"ContainerDied","Data":"96fb3d5b49793e6a5ac7508368b0bc678a6dd66a4a046aa09fbf4c346a0b3c85"}
Nov 03 16:03:22.111835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:22.111448    4964 scope.go:122] "RemoveContainer" containerID="2923eeb96076e7d1b51c7bd922fa97d5a8c6bcbe9e8f187ca4a189d488e9df3c"
Nov 03 16:03:22.134612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:22.134584    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b94f5bc97-w9xjm"]
Nov 03 16:03:22.139582 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:22.139555    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b94f5bc97-w9xjm"]
Nov 03 16:03:22.610077 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:22.609891    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" path="/var/lib/kubelet/pods/17eca28e-8ea1-4193-92c1-32dd16b78e0d/volumes"
Nov 03 16:03:32.565585 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:03:32.565544    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:03:34.166935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:34.166895    4964 generic.go:356] "Generic (PLEG): container finished" podID="a71be33d-a6e5-4b3a-b703-64cc2f214f5b" containerID="ccec6db0c1cf30f16ff022dc8d2d6b14a1712e9f690d6f040cc06b70804f823a" exitCode=0
Nov 03 16:03:34.167400 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:34.166970    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb" event={"ID":"a71be33d-a6e5-4b3a-b703-64cc2f214f5b","Type":"ContainerDied","Data":"ccec6db0c1cf30f16ff022dc8d2d6b14a1712e9f690d6f040cc06b70804f823a"}
Nov 03 16:03:34.167400 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:34.167318    4964 scope.go:122] "RemoveContainer" containerID="ccec6db0c1cf30f16ff022dc8d2d6b14a1712e9f690d6f040cc06b70804f823a"
Nov 03 16:03:35.173055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:03:35.173020    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9sxgb" event={"ID":"a71be33d-a6e5-4b3a-b703-64cc2f214f5b","Type":"ContainerStarted","Data":"916eb811a0affa7319ff33ce92d7153ac52f207ba420c3888978fac31028d3c1"}
Nov 03 16:04:01.280641 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:01.280605    4964 generic.go:356] "Generic (PLEG): container finished" podID="4cb7e596-9c64-4bcc-8841-45d1fd5e0485" containerID="085a2324a2130420251d229ef543af194bc127886a1689b679f17cb2a9f5d150" exitCode=0
Nov 03 16:04:01.281081 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:01.280687    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-49czb-t8nxv" event={"ID":"4cb7e596-9c64-4bcc-8841-45d1fd5e0485","Type":"ContainerDied","Data":"085a2324a2130420251d229ef543af194bc127886a1689b679f17cb2a9f5d150"}
Nov 03 16:04:02.441970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.441942    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:04:02.546297 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546209    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle\") pod \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") "
Nov 03 16:04:02.546468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546319    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle\") pod \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") "
Nov 03 16:04:02.546468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546354    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9\" (UniqueName: \"kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9\") pod \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") "
Nov 03 16:04:02.546468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546453    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path\") pod \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\" (UID: \"4cb7e596-9c64-4bcc-8841-45d1fd5e0485\") "
Nov 03 16:04:02.546680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546658    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle" pod "4cb7e596-9c64-4bcc-8841-45d1fd5e0485" (UID: "4cb7e596-9c64-4bcc-8841-45d1fd5e0485"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:04:02.546725 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546682    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle" pod "4cb7e596-9c64-4bcc-8841-45d1fd5e0485" (UID: "4cb7e596-9c64-4bcc-8841-45d1fd5e0485"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:04:02.546946 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.546881    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path" pod "4cb7e596-9c64-4bcc-8841-45d1fd5e0485" (UID: "4cb7e596-9c64-4bcc-8841-45d1fd5e0485"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:04:02.548421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.548375    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9" pod "4cb7e596-9c64-4bcc-8841-45d1fd5e0485" (UID: "4cb7e596-9c64-4bcc-8841-45d1fd5e0485"). InnerVolumeSpecName "kube-api-access-j9rj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:04:02.565610 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:04:02.565573    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:04:02.647626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.647584    4964 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-service-ca-bundle\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:04:02.647626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.647624    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rj9\" (UniqueName: \"kubernetes.io/projected/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-kube-api-access-j9rj9\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:04:02.647626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.647641    4964 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-archives-path\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:04:02.647910 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:02.647657    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb7e596-9c64-4bcc-8841-45d1fd5e0485-trusted-ca-bundle\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:04:03.289972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:03.289942    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-49czb-t8nxv"
Nov 03 16:04:03.290170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:03.289935    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-49czb-t8nxv" event={"ID":"4cb7e596-9c64-4bcc-8841-45d1fd5e0485","Type":"ContainerDied","Data":"d3429c4dbfcd807133900ea719abce77c3d7b4d1ab5b15a7eb91ebdb6ebc078e"}
Nov 03 16:04:03.290170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:04:03.290064    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3429c4dbfcd807133900ea719abce77c3d7b4d1ab5b15a7eb91ebdb6ebc078e"
Nov 03 16:04:32.566299 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:04:32.566212    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:05:02.566093 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:05:02.566055    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:05:32.566037 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:05:32.566000    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:06:02.565117 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:06:02.565033    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:06:32.565935 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:06:32.565899    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:07:02.565052 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:07:02.565009    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:07:18.634714 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:07:18.634687    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:07:18.636788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:07:18.636765    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:07:32.565987 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:07:32.565907    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:08:02.566152 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:08:02.566113    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:08:32.566112 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:08:32.566073    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:09:02.565911 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:09:02.565830    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:09:18.632414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:09:18.632367    4964 scope.go:122] "RemoveContainer" containerID="750d90006b95b446559274619a02c24158258f482ffa0f97a2f697ff1aac1b11"
Nov 03 16:09:32.566420 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:09:32.566366    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:10:02.566750 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:10:02.566709    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:10:32.567596 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:10:32.567501    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:10:35.631491 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631448    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"]
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631795    4964 cpu_manager.go:414] "containerMap: removing container" podUID="4cb7e596-9c64-4bcc-8841-45d1fd5e0485" containerName="insights-gathering"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631811    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb7e596-9c64-4bcc-8841-45d1fd5e0485" containerName="insights-gathering"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631844    4964 cpu_manager.go:414] "containerMap: removing container" podUID="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" containerName="insights-gathering"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631851    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" containerName="insights-gathering"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631862    4964 cpu_manager.go:414] "containerMap: removing container" podUID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" containerName="registry"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631869    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" containerName="registry"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631880    4964 cpu_manager.go:414] "containerMap: removing container" podUID="df53f549-e19a-4304-8387-4afa8528b975" containerName="registry"
Nov 03 16:10:35.631936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631888    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df53f549-e19a-4304-8387-4afa8528b975" containerName="registry"
Nov 03 16:10:35.632277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631983    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb7e596-9c64-4bcc-8841-45d1fd5e0485" containerName="insights-gathering"
Nov 03 16:10:35.632277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.631995    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a088fef-29ab-4b9e-a90d-f6560aa4ee43" containerName="insights-gathering"
Nov 03 16:10:35.632277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.632005    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="17eca28e-8ea1-4193-92c1-32dd16b78e0d" containerName="registry"
Nov 03 16:10:35.632277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.632013    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="df53f549-e19a-4304-8387-4afa8528b975" containerName="registry"
Nov 03 16:10:35.635430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.635402    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.648543 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.648519    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 16:10:35.648693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.648645    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-gsshq\""
Nov 03 16:10:35.652531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.652501    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"]
Nov 03 16:10:35.748476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748431    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748479    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5175180-6adf-411c-ab88-b709371d07ba-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748513    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748534    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87d9\" (UniqueName: \"kubernetes.io/projected/c5175180-6adf-411c-ab88-b709371d07ba-kube-api-access-w87d9\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748629    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748715    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748762    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748790    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5175180-6adf-411c-ab88-b709371d07ba-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.748959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.748833    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5175180-6adf-411c-ab88-b709371d07ba-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.849729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849684    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.849934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849750    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5175180-6adf-411c-ab88-b709371d07ba-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.849934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849827    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.849934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849861    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87d9\" (UniqueName: \"kubernetes.io/projected/c5175180-6adf-411c-ab88-b709371d07ba-kube-api-access-w87d9\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.849934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849902    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849943    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.849974    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850007    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5175180-6adf-411c-ab88-b709371d07ba-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850034    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5175180-6adf-411c-ab88-b709371d07ba-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850359 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850198    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850359 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850320    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850359 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850322    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850510    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5175180-6adf-411c-ab88-b709371d07ba-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.850611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.850553    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.852344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.852323    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5175180-6adf-411c-ab88-b709371d07ba-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.852649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.852633    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5175180-6adf-411c-ab88-b709371d07ba-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.862519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.862493    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5175180-6adf-411c-ab88-b709371d07ba-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.862767 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.862743    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87d9\" (UniqueName: \"kubernetes.io/projected/c5175180-6adf-411c-ab88-b709371d07ba-kube-api-access-w87d9\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr\" (UID: \"c5175180-6adf-411c-ab88-b709371d07ba\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:35.949501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:35.949469    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:36.232240 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:36.232160    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"]
Nov 03 16:10:36.235173 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:10:36.235148    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5175180_6adf_411c_ab88_b709371d07ba.slice/crio-3c775d4dd69b58416186beeee32ee06a22f1162f01dab708d6e394444256fc00 WatchSource:0}: Error finding container 3c775d4dd69b58416186beeee32ee06a22f1162f01dab708d6e394444256fc00: Status 404 returned error can't find the container with id 3c775d4dd69b58416186beeee32ee06a22f1162f01dab708d6e394444256fc00
Nov 03 16:10:36.237151 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:36.237132    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:10:36.906199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:36.906159    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr" event={"ID":"c5175180-6adf-411c-ab88-b709371d07ba","Type":"ContainerStarted","Data":"3c775d4dd69b58416186beeee32ee06a22f1162f01dab708d6e394444256fc00"}
Nov 03 16:10:39.069419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:39.069363    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:10:39.069782 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:39.069454    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:10:39.069782 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:39.069486    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:10:39.920797 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:39.920752    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr" event={"ID":"c5175180-6adf-411c-ab88-b709371d07ba","Type":"ContainerStarted","Data":"c0a991857ef09fb10a12bd13dd9e5ea8ccd161599964f0c500eae07b4b8b608b"}
Nov 03 16:10:39.948117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:39.948056    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr" podStartSLOduration=2.116963215 podStartE2EDuration="4.948039556s" totalImagesPullingTime="2.831076341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:39.946246148 +0000 UTC m=+501.459627926" watchObservedRunningTime="2026-11-03 16:10:39.948039556 +0000 UTC m=+501.461421329"
Nov 03 16:10:39.950538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:39.950511    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:40.955086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:40.955048    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:41.929927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:41.929894    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:10:41.930839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:10:41.930821    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr"
Nov 03 16:11:02.565926 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:11:02.565881    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:11:12.517015 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.516977    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"]
Nov 03 16:11:12.520427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.520405    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.549563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.549522    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"]
Nov 03 16:11:12.590246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.590211    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 16:11:12.590494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.590473    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 16:11:12.590914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.590898    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 16:11:12.590968 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.590930    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 16:11:12.591009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.590971    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 16:11:12.591049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.591004    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-f626w\""
Nov 03 16:11:12.662300 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.662261    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqwm\" (UniqueName: \"kubernetes.io/projected/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-kube-api-access-kpqwm\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.662502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.662312    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.662502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.662333    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.662502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.662486    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.662609 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.662516    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.662609 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.662540    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.763480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.763433    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqwm\" (UniqueName: \"kubernetes.io/projected/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-kube-api-access-kpqwm\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.763699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.763499    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.763699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.763529    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.763699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.763576    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.763699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.763619    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.763699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.763655    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.764330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.764302    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.766101 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.766068    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.766300 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.766279    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.766362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.766311    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.766362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.766341    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.777222 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.777147    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqwm\" (UniqueName: \"kubernetes.io/projected/7c4b613c-7266-4dc0-8bc6-f76cc75947c7-kube-api-access-kpqwm\") pod \"opendatahub-operator-controller-manager-d59cb86d5-6q2wx\" (UID: \"7c4b613c-7266-4dc0-8bc6-f76cc75947c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:12.835285 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:12.835242    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:13.141276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:13.141248    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"]
Nov 03 16:11:13.143145 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:13.143106    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4b613c_7266_4dc0_8bc6_f76cc75947c7.slice/crio-3560b0a5e11f1d8b26295f1a9c5ba1c6349dbacd18db68ea1ef58df528d91ba3 WatchSource:0}: Error finding container 3560b0a5e11f1d8b26295f1a9c5ba1c6349dbacd18db68ea1ef58df528d91ba3: Status 404 returned error can't find the container with id 3560b0a5e11f1d8b26295f1a9c5ba1c6349dbacd18db68ea1ef58df528d91ba3
Nov 03 16:11:14.067919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:14.067803    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx" event={"ID":"7c4b613c-7266-4dc0-8bc6-f76cc75947c7","Type":"ContainerStarted","Data":"3560b0a5e11f1d8b26295f1a9c5ba1c6349dbacd18db68ea1ef58df528d91ba3"}
Nov 03 16:11:17.084744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:17.084661    4964 generic.go:356] "Generic (PLEG): container finished" podID="7c4b613c-7266-4dc0-8bc6-f76cc75947c7" containerID="6beb4b6068292db29fc27e7a048c42a67c5c94747d1ef6f0f0953d98b4bcc48e" exitCode=0
Nov 03 16:11:17.084744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:17.084724    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx" event={"ID":"7c4b613c-7266-4dc0-8bc6-f76cc75947c7","Type":"ContainerDied","Data":"6beb4b6068292db29fc27e7a048c42a67c5c94747d1ef6f0f0953d98b4bcc48e"}
Nov 03 16:11:18.091551 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:18.091502    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx" event={"ID":"7c4b613c-7266-4dc0-8bc6-f76cc75947c7","Type":"ContainerStarted","Data":"03662d0258bb7cbf2c6d150c926fabb60eb819f3e76d9e5f5b11c6bf8327f287"}
Nov 03 16:11:18.092038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:18.091613    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:18.138851 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:18.138787    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx" podStartSLOduration=2.267782762 podStartE2EDuration="6.13877029s" totalImagesPullingTime="3.293700527s" totalInitContainerRuntime="577.287001ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:18.136777275 +0000 UTC m=+539.650159051" watchObservedRunningTime="2026-11-03 16:11:18.13877029 +0000 UTC m=+539.652152068"
Nov 03 16:11:29.098439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:29.098400    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-6q2wx"
Nov 03 16:11:32.566146 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:11:32.566106    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:11:35.498895 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.498855    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"]
Nov 03 16:11:35.505493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.505471    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:35.510117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.510090    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p8v6q\""
Nov 03 16:11:35.513009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.512982    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 16:11:35.513009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.512985    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 16:11:35.520944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.520912    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"]
Nov 03 16:11:35.658095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.658053    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrl8p\" (UniqueName: \"kubernetes.io/projected/80ede112-ea2e-4d99-b077-c81612f2429b-kube-api-access-hrl8p\") pod \"limitador-operator-controller-manager-5778bb4449-r27kw\" (UID: \"80ede112-ea2e-4d99-b077-c81612f2429b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:35.759441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.759331    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrl8p\" (UniqueName: \"kubernetes.io/projected/80ede112-ea2e-4d99-b077-c81612f2429b-kube-api-access-hrl8p\") pod \"limitador-operator-controller-manager-5778bb4449-r27kw\" (UID: \"80ede112-ea2e-4d99-b077-c81612f2429b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:35.776033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.775995    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrl8p\" (UniqueName: \"kubernetes.io/projected/80ede112-ea2e-4d99-b077-c81612f2429b-kube-api-access-hrl8p\") pod \"limitador-operator-controller-manager-5778bb4449-r27kw\" (UID: \"80ede112-ea2e-4d99-b077-c81612f2429b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:35.819948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:35.819898    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:36.103247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:36.103107    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"]
Nov 03 16:11:36.106445 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:36.106412    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ede112_ea2e_4d99_b077_c81612f2429b.slice/crio-855d8f8366a48f21dad6658447c81e14632636373386e3ce10ccb3d6776d2d9e WatchSource:0}: Error finding container 855d8f8366a48f21dad6658447c81e14632636373386e3ce10ccb3d6776d2d9e: Status 404 returned error can't find the container with id 855d8f8366a48f21dad6658447c81e14632636373386e3ce10ccb3d6776d2d9e
Nov 03 16:11:36.175233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:36.175200    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw" event={"ID":"80ede112-ea2e-4d99-b077-c81612f2429b","Type":"ContainerStarted","Data":"855d8f8366a48f21dad6658447c81e14632636373386e3ce10ccb3d6776d2d9e"}
Nov 03 16:11:39.037154 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.037117    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"]
Nov 03 16:11:39.108752 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.108664    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"]
Nov 03 16:11:39.108925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.108807    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:39.114074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.114050    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 16:11:39.114242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.114230    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rdh79\""
Nov 03 16:11:39.291712 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.291622    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvbm\" (UniqueName: \"kubernetes.io/projected/4deed5e7-c8e6-4007-9dd0-5b1550dc53b6-kube-api-access-7lvbm\") pod \"dns-operator-controller-manager-8557cbdb84-28k8r\" (UID: \"4deed5e7-c8e6-4007-9dd0-5b1550dc53b6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:39.393170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.393126    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvbm\" (UniqueName: \"kubernetes.io/projected/4deed5e7-c8e6-4007-9dd0-5b1550dc53b6-kube-api-access-7lvbm\") pod \"dns-operator-controller-manager-8557cbdb84-28k8r\" (UID: \"4deed5e7-c8e6-4007-9dd0-5b1550dc53b6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:39.406467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.406430    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvbm\" (UniqueName: \"kubernetes.io/projected/4deed5e7-c8e6-4007-9dd0-5b1550dc53b6-kube-api-access-7lvbm\") pod \"dns-operator-controller-manager-8557cbdb84-28k8r\" (UID: \"4deed5e7-c8e6-4007-9dd0-5b1550dc53b6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:39.440223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.440184    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:39.722680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:39.722614    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"]
Nov 03 16:11:39.725592 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:39.725553    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deed5e7_c8e6_4007_9dd0_5b1550dc53b6.slice/crio-9dfb4e9c3220a29f6e67a34ebfcf61a4bbb45d7e7550b220abcc9e6b6e3b4cca WatchSource:0}: Error finding container 9dfb4e9c3220a29f6e67a34ebfcf61a4bbb45d7e7550b220abcc9e6b6e3b4cca: Status 404 returned error can't find the container with id 9dfb4e9c3220a29f6e67a34ebfcf61a4bbb45d7e7550b220abcc9e6b6e3b4cca
Nov 03 16:11:40.194871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:40.194832    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw" event={"ID":"80ede112-ea2e-4d99-b077-c81612f2429b","Type":"ContainerStarted","Data":"697ffd820b81f1f21969377f4f5afb306a809732774c78b99a3c8fd639f3f20e"}
Nov 03 16:11:40.195382 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:40.194943    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:40.196199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:40.196174    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r" event={"ID":"4deed5e7-c8e6-4007-9dd0-5b1550dc53b6","Type":"ContainerStarted","Data":"9dfb4e9c3220a29f6e67a34ebfcf61a4bbb45d7e7550b220abcc9e6b6e3b4cca"}
Nov 03 16:11:40.219461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:40.219379    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw" podStartSLOduration=1.8000713780000002 podStartE2EDuration="5.219363665s" totalImagesPullingTime="3.419292287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:40.218111297 +0000 UTC m=+561.731493073" watchObservedRunningTime="2026-11-03 16:11:40.219363665 +0000 UTC m=+561.732745434"
Nov 03 16:11:42.214475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:42.214432    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r" event={"ID":"4deed5e7-c8e6-4007-9dd0-5b1550dc53b6","Type":"ContainerStarted","Data":"cc7408c6e3418ddc3fa2e90f51ea2acde2e617c4ff7e2a781003228616a0bc61"}
Nov 03 16:11:42.214891 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:42.214646    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:42.237058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:42.237003    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r" podStartSLOduration=0.93423201 podStartE2EDuration="3.236990645s" totalImagesPullingTime="2.302758635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:42.235229612 +0000 UTC m=+563.748611389" watchObservedRunningTime="2026-11-03 16:11:42.236990645 +0000 UTC m=+563.750372421"
Nov 03 16:11:43.013475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.013440    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"]
Nov 03 16:11:43.016908 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.016888    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:43.026841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.026812    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-48jgd\""
Nov 03 16:11:43.033556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.033529    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"]
Nov 03 16:11:43.126100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.126049    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6sg\" (UniqueName: \"kubernetes.io/projected/8cbdc4a1-7e44-4183-9a1c-a51b22a69623-kube-api-access-vs6sg\") pod \"authorino-operator-7b7bb4bc77-9vc9h\" (UID: \"8cbdc4a1-7e44-4183-9a1c-a51b22a69623\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:43.227250 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.227209    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6sg\" (UniqueName: \"kubernetes.io/projected/8cbdc4a1-7e44-4183-9a1c-a51b22a69623-kube-api-access-vs6sg\") pod \"authorino-operator-7b7bb4bc77-9vc9h\" (UID: \"8cbdc4a1-7e44-4183-9a1c-a51b22a69623\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:43.248883 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.248850    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6sg\" (UniqueName: \"kubernetes.io/projected/8cbdc4a1-7e44-4183-9a1c-a51b22a69623-kube-api-access-vs6sg\") pod \"authorino-operator-7b7bb4bc77-9vc9h\" (UID: \"8cbdc4a1-7e44-4183-9a1c-a51b22a69623\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:43.329847 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.329753    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:43.614492 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:43.614465    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"]
Nov 03 16:11:43.617135 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:43.617104    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbdc4a1_7e44_4183_9a1c_a51b22a69623.slice/crio-7207d249583ff4ec37d5a77130d60d923d2bd01dec4683fa8309dc40ae56ad09 WatchSource:0}: Error finding container 7207d249583ff4ec37d5a77130d60d923d2bd01dec4683fa8309dc40ae56ad09: Status 404 returned error can't find the container with id 7207d249583ff4ec37d5a77130d60d923d2bd01dec4683fa8309dc40ae56ad09
Nov 03 16:11:44.225800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:44.225764    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h" event={"ID":"8cbdc4a1-7e44-4183-9a1c-a51b22a69623","Type":"ContainerStarted","Data":"7207d249583ff4ec37d5a77130d60d923d2bd01dec4683fa8309dc40ae56ad09"}
Nov 03 16:11:46.237555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:46.237521    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h" event={"ID":"8cbdc4a1-7e44-4183-9a1c-a51b22a69623","Type":"ContainerStarted","Data":"200a92e0001c384d1bb24695d4b7f8961712c17b6ea0dcd58cbb43e33848d707"}
Nov 03 16:11:46.238011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:46.237639    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:46.275613 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:46.275543    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h" podStartSLOduration=2.407538121 podStartE2EDuration="4.275525497s" totalImagesPullingTime="1.867987376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:46.274507492 +0000 UTC m=+567.787889278" watchObservedRunningTime="2026-11-03 16:11:46.275525497 +0000 UTC m=+567.788907271"
Nov 03 16:11:48.216758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.216724    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"]
Nov 03 16:11:48.220242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.220222    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.227226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.227124    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9dbgw\""
Nov 03 16:11:48.233627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.233598    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"]
Nov 03 16:11:48.269535 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.269500    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpttn\" (UniqueName: \"kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn\") pod \"kuadrant-operator-controller-manager-678d87766f-bj9c9\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.269694 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.269560    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bj9c9\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.370911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.370867    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpttn\" (UniqueName: \"kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn\") pod \"kuadrant-operator-controller-manager-678d87766f-bj9c9\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.371062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.370928    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bj9c9\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.371281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.371266    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bj9c9\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.384780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.384747    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpttn\" (UniqueName: \"kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn\") pod \"kuadrant-operator-controller-manager-678d87766f-bj9c9\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.534121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.534024    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:48.817579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:48.817547    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"]
Nov 03 16:11:48.822589 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:48.822554    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42924e66_4865_4cd3_9771_4e819f4bacb2.slice/crio-6c0a7f8251c52af8f89b6557bdbbe92780dd2784d2395fd487e65b2f51b243a0 WatchSource:0}: Error finding container 6c0a7f8251c52af8f89b6557bdbbe92780dd2784d2395fd487e65b2f51b243a0: Status 404 returned error can't find the container with id 6c0a7f8251c52af8f89b6557bdbbe92780dd2784d2395fd487e65b2f51b243a0
Nov 03 16:11:49.252427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:49.252380    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9" event={"ID":"42924e66-4865-4cd3-9771-4e819f4bacb2","Type":"ContainerStarted","Data":"6c0a7f8251c52af8f89b6557bdbbe92780dd2784d2395fd487e65b2f51b243a0"}
Nov 03 16:11:51.203473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:51.203437    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-r27kw"
Nov 03 16:11:53.220911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:53.220884    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-28k8r"
Nov 03 16:11:53.282740 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:53.282695    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9" event={"ID":"42924e66-4865-4cd3-9771-4e819f4bacb2","Type":"ContainerStarted","Data":"d15eff275e1337f773f9bd858d99fcc1d40cf2ed37e019d2dc07d9ccdebdcba5"}
Nov 03 16:11:53.282957 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:53.282806    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:11:53.307324 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:53.307266    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9" podStartSLOduration=0.935959177 podStartE2EDuration="5.307247114s" totalImagesPullingTime="4.371287937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:53.30692847 +0000 UTC m=+574.820310247" watchObservedRunningTime="2026-11-03 16:11:53.307247114 +0000 UTC m=+574.820628892"
Nov 03 16:11:54.841133 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.841091    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"]
Nov 03 16:11:54.852742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.852710    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.861833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.861805    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tjqfj\""
Nov 03 16:11:54.863191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.863156    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"]
Nov 03 16:11:54.927815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.927777    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qfv\" (UniqueName: \"kubernetes.io/projected/657478d3-04a7-4e89-9059-14c746334109-kube-api-access-w8qfv\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.927999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.927837    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657478d3-04a7-4e89-9059-14c746334109-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.927999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.927902    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.927999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.927919    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.927999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.927934    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657478d3-04a7-4e89-9059-14c746334109-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.927999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.927968    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.928191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.928004    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.928191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.928045    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657478d3-04a7-4e89-9059-14c746334109-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:54.928191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:54.928066    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.028718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.028676    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfv\" (UniqueName: \"kubernetes.io/projected/657478d3-04a7-4e89-9059-14c746334109-kube-api-access-w8qfv\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.028922 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.028752    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657478d3-04a7-4e89-9059-14c746334109-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.028922 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.028814    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.028931    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.028967    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657478d3-04a7-4e89-9059-14c746334109-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029019    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029044    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029089    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657478d3-04a7-4e89-9059-14c746334109-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029222    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029270    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029351    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029437    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029561    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.029682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.029662    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657478d3-04a7-4e89-9059-14c746334109-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.031183 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.031160    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657478d3-04a7-4e89-9059-14c746334109-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.031457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.031441    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657478d3-04a7-4e89-9059-14c746334109-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.044224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.044188    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657478d3-04a7-4e89-9059-14c746334109-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.044356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.044288    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qfv\" (UniqueName: \"kubernetes.io/projected/657478d3-04a7-4e89-9059-14c746334109-kube-api-access-w8qfv\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6fwq55k\" (UID: \"657478d3-04a7-4e89-9059-14c746334109\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.168887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.168848    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:55.458280 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.458056    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"]
Nov 03 16:11:55.460935 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:55.460907    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657478d3_04a7_4e89_9059_14c746334109.slice/crio-ce18b26cdd620cd7d42453c011b3ee0a9bc0e6bbe56e7eccabf0c1fd488e5f49 WatchSource:0}: Error finding container ce18b26cdd620cd7d42453c011b3ee0a9bc0e6bbe56e7eccabf0c1fd488e5f49: Status 404 returned error can't find the container with id ce18b26cdd620cd7d42453c011b3ee0a9bc0e6bbe56e7eccabf0c1fd488e5f49
Nov 03 16:11:55.463175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.463140    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:11:55.463254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.463217    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:11:55.463254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:55.463247    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:11:56.302093 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.302054    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k" event={"ID":"657478d3-04a7-4e89-9059-14c746334109","Type":"ContainerStarted","Data":"0deda65178b2072625310c53aceac7be7791b927ababd82f050d83e1a07ab046"}
Nov 03 16:11:56.302566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.302108    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k" event={"ID":"657478d3-04a7-4e89-9059-14c746334109","Type":"ContainerStarted","Data":"ce18b26cdd620cd7d42453c011b3ee0a9bc0e6bbe56e7eccabf0c1fd488e5f49"}
Nov 03 16:11:56.329550 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.329493    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k" podStartSLOduration=2.329478431 podStartE2EDuration="2.329478431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:56.328008452 +0000 UTC m=+577.841390226" watchObservedRunningTime="2026-11-03 16:11:56.329478431 +0000 UTC m=+577.842860206"
Nov 03 16:11:56.898639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.898604    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"]
Nov 03 16:11:56.902453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.902427    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:56.925481 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.925446    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-fwrrp\""
Nov 03 16:11:56.925660 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.925614    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"]
Nov 03 16:11:56.925951 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.925932    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 16:11:56.926133 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.926118    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 16:11:56.930312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.930289    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 16:11:56.946289 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.946257    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/740433eb-df94-4702-beaa-26a4fad6e366-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:56.946484 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.946299    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh6w\" (UniqueName: \"kubernetes.io/projected/740433eb-df94-4702-beaa-26a4fad6e366-kube-api-access-lnh6w\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:56.946484 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.946467    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740433eb-df94-4702-beaa-26a4fad6e366-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:56.946571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:56.946506    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740433eb-df94-4702-beaa-26a4fad6e366-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.047897 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.047863    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740433eb-df94-4702-beaa-26a4fad6e366-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.048067 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.047910    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/740433eb-df94-4702-beaa-26a4fad6e366-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.048067 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.048027    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh6w\" (UniqueName: \"kubernetes.io/projected/740433eb-df94-4702-beaa-26a4fad6e366-kube-api-access-lnh6w\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.048178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.048160    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740433eb-df94-4702-beaa-26a4fad6e366-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.048267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.048249    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740433eb-df94-4702-beaa-26a4fad6e366-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.048525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.048508    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740433eb-df94-4702-beaa-26a4fad6e366-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.048779 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.048761    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/740433eb-df94-4702-beaa-26a4fad6e366-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.060201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.060174    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh6w\" (UniqueName: \"kubernetes.io/projected/740433eb-df94-4702-beaa-26a4fad6e366-kube-api-access-lnh6w\") pod \"kserve-module-controller-manager-7566cf5fc8-lfxrp\" (UID: \"740433eb-df94-4702-beaa-26a4fad6e366\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.169900 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.169865    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:57.174838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.174814    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:57.215474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.215436    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:11:57.244613 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.244585    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9vc9h"
Nov 03 16:11:57.308282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.308252    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:57.309436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.309414    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6fwq55k"
Nov 03 16:11:57.494257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:57.494180    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"]
Nov 03 16:11:57.497308 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:11:57.497279    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740433eb_df94_4702_beaa_26a4fad6e366.slice/crio-6a4e25029af745475a17bb78e51ed96c058f3450a112aadaee4865340b3873b3 WatchSource:0}: Error finding container 6a4e25029af745475a17bb78e51ed96c058f3450a112aadaee4865340b3873b3: Status 404 returned error can't find the container with id 6a4e25029af745475a17bb78e51ed96c058f3450a112aadaee4865340b3873b3
Nov 03 16:11:58.315680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:11:58.315636    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp" event={"ID":"740433eb-df94-4702-beaa-26a4fad6e366","Type":"ContainerStarted","Data":"6a4e25029af745475a17bb78e51ed96c058f3450a112aadaee4865340b3873b3"}
Nov 03 16:12:00.327633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:00.327597    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp" event={"ID":"740433eb-df94-4702-beaa-26a4fad6e366","Type":"ContainerStarted","Data":"40f9848ccff1ca89d7deb3bdb2f613ddc5a2fcfbee2219a12a620299abc03d78"}
Nov 03 16:12:00.328080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:00.327774    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:12:00.351125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:00.351068    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp" podStartSLOduration=1.997261982 podStartE2EDuration="4.351051954s" totalImagesPullingTime="2.353789972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:00.349541597 +0000 UTC m=+581.862923373" watchObservedRunningTime="2026-11-03 16:12:00.351051954 +0000 UTC m=+581.864433747"
Nov 03 16:12:02.565810 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:12:02.565698    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:12:04.290916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:04.290884    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:12:18.664566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:18.664539    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:12:18.665893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:18.665870    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:12:21.333166 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:21.333128    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp"
Nov 03 16:12:32.566033 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:12:32.565991    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:12:51.045215 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.045174    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"]
Nov 03 16:12:51.055573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.055534    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.066083 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.066049    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"]
Nov 03 16:12:51.082405 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.082365    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 16:12:51.082572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.082477    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-x8gtx\""
Nov 03 16:12:51.151857 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.151814    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmn2\" (UniqueName: \"kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2\") pod \"llmisvc-controller-manager-869d5dcc5d-nk5ck\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.151857 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.151860    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nk5ck\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.253349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.253305    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmn2\" (UniqueName: \"kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2\") pod \"llmisvc-controller-manager-869d5dcc5d-nk5ck\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.253349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.253349    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nk5ck\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.255855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.255834    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nk5ck\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.273658 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.273625    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmn2\" (UniqueName: \"kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2\") pod \"llmisvc-controller-manager-869d5dcc5d-nk5ck\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.374527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.374439    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:51.660332 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:51.660287    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"]
Nov 03 16:12:51.678307 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:12:51.678276    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod44580bb4_9d97_4835_8175_103a37b4a2ae.slice/crio-e0183ee36fea5f5793e24c79a4126cf4a9e574b997c853ee4813e8a7f5db6f01 WatchSource:0}: Error finding container e0183ee36fea5f5793e24c79a4126cf4a9e574b997c853ee4813e8a7f5db6f01: Status 404 returned error can't find the container with id e0183ee36fea5f5793e24c79a4126cf4a9e574b997c853ee4813e8a7f5db6f01
Nov 03 16:12:52.580452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:52.580414    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck" event={"ID":"44580bb4-9d97-4835-8175-103a37b4a2ae","Type":"ContainerStarted","Data":"e0183ee36fea5f5793e24c79a4126cf4a9e574b997c853ee4813e8a7f5db6f01"}
Nov 03 16:12:54.592320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:54.592281    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck" event={"ID":"44580bb4-9d97-4835-8175-103a37b4a2ae","Type":"ContainerStarted","Data":"ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101"}
Nov 03 16:12:54.592780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:54.592404    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:12:57.519542 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.519486    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck" podStartSLOduration=4.545136224 podStartE2EDuration="6.519469391s" totalImagesPullingTime="1.974333167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:54.629487522 +0000 UTC m=+636.142869312" watchObservedRunningTime="2026-11-03 16:12:57.519469391 +0000 UTC m=+639.032851171"
Nov 03 16:12:57.519948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.519737    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:12:57.523344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.523326    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.533020 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.532881    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wdnhl\""
Nov 03 16:12:57.533020 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.532992    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 16:12:57.534988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.534952    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:12:57.574270 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.574228    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:12:57.619246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.619211    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/003233e0-3e53-410b-90f2-9881e337aa81-config-file\") pod \"limitador-limitador-7c9b87d574-jpjhq\" (UID: \"003233e0-3e53-410b-90f2-9881e337aa81\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.619450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.619254    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd4l\" (UniqueName: \"kubernetes.io/projected/003233e0-3e53-410b-90f2-9881e337aa81-kube-api-access-gnd4l\") pod \"limitador-limitador-7c9b87d574-jpjhq\" (UID: \"003233e0-3e53-410b-90f2-9881e337aa81\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.720360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.720321    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/003233e0-3e53-410b-90f2-9881e337aa81-config-file\") pod \"limitador-limitador-7c9b87d574-jpjhq\" (UID: \"003233e0-3e53-410b-90f2-9881e337aa81\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.720360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.720364    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd4l\" (UniqueName: \"kubernetes.io/projected/003233e0-3e53-410b-90f2-9881e337aa81-kube-api-access-gnd4l\") pod \"limitador-limitador-7c9b87d574-jpjhq\" (UID: \"003233e0-3e53-410b-90f2-9881e337aa81\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.721014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.720993    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/003233e0-3e53-410b-90f2-9881e337aa81-config-file\") pod \"limitador-limitador-7c9b87d574-jpjhq\" (UID: \"003233e0-3e53-410b-90f2-9881e337aa81\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.732883 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.732850    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd4l\" (UniqueName: \"kubernetes.io/projected/003233e0-3e53-410b-90f2-9881e337aa81-kube-api-access-gnd4l\") pod \"limitador-limitador-7c9b87d574-jpjhq\" (UID: \"003233e0-3e53-410b-90f2-9881e337aa81\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:57.837204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:57.837113    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:12:58.115504 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:58.115481    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:12:58.117689 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:12:58.117658    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003233e0_3e53_410b_90f2_9881e337aa81.slice/crio-4a111a8f3724ea969376854b5b6da282346fb6eef9315ae8b354c83de378391c WatchSource:0}: Error finding container 4a111a8f3724ea969376854b5b6da282346fb6eef9315ae8b354c83de378391c: Status 404 returned error can't find the container with id 4a111a8f3724ea969376854b5b6da282346fb6eef9315ae8b354c83de378391c
Nov 03 16:12:58.611131 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:12:58.611091    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq" event={"ID":"003233e0-3e53-410b-90f2-9881e337aa81","Type":"ContainerStarted","Data":"4a111a8f3724ea969376854b5b6da282346fb6eef9315ae8b354c83de378391c"}
Nov 03 16:13:00.626995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:13:00.626954    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq" event={"ID":"003233e0-3e53-410b-90f2-9881e337aa81","Type":"ContainerStarted","Data":"28ca4cbffc3cbc3806e514094031ee4bb99444e2b59019a6db6a874effe4c3b1"}
Nov 03 16:13:00.627545 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:13:00.627037    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:13:00.649993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:13:00.649903    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq" podStartSLOduration=1.971670144 podStartE2EDuration="3.649882532s" totalImagesPullingTime="1.678212388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:00.64685768 +0000 UTC m=+642.160239457" watchObservedRunningTime="2026-11-03 16:13:00.649882532 +0000 UTC m=+642.163264312"
Nov 03 16:13:02.569353 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:13:02.569317    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:13:11.633796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:13:11.633764    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"
Nov 03 16:13:25.599097 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:13:25.599066    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:13:32.564897 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:13:32.564809    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:14:02.566125 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:14:02.566085    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:14:10.262801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:10.262763    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:14:22.916796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:22.916760    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"]
Nov 03 16:14:22.920623 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:22.920604    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:22.930108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:22.930081    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"]
Nov 03 16:14:22.979455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:22.979415    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f59614-ca2f-4f2d-ac69-699714a2f4c5-cert\") pod \"llmisvc-controller-manager-6466c554d8-cljjs\" (UID: \"25f59614-ca2f-4f2d-ac69-699714a2f4c5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:22.979653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:22.979478    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clxng\" (UniqueName: \"kubernetes.io/projected/25f59614-ca2f-4f2d-ac69-699714a2f4c5-kube-api-access-clxng\") pod \"llmisvc-controller-manager-6466c554d8-cljjs\" (UID: \"25f59614-ca2f-4f2d-ac69-699714a2f4c5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:23.080958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:23.080916    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f59614-ca2f-4f2d-ac69-699714a2f4c5-cert\") pod \"llmisvc-controller-manager-6466c554d8-cljjs\" (UID: \"25f59614-ca2f-4f2d-ac69-699714a2f4c5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:23.081200 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:23.080989    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clxng\" (UniqueName: \"kubernetes.io/projected/25f59614-ca2f-4f2d-ac69-699714a2f4c5-kube-api-access-clxng\") pod \"llmisvc-controller-manager-6466c554d8-cljjs\" (UID: \"25f59614-ca2f-4f2d-ac69-699714a2f4c5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:23.083497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:23.083476    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f59614-ca2f-4f2d-ac69-699714a2f4c5-cert\") pod \"llmisvc-controller-manager-6466c554d8-cljjs\" (UID: \"25f59614-ca2f-4f2d-ac69-699714a2f4c5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:23.093707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:23.093680    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clxng\" (UniqueName: \"kubernetes.io/projected/25f59614-ca2f-4f2d-ac69-699714a2f4c5-kube-api-access-clxng\") pod \"llmisvc-controller-manager-6466c554d8-cljjs\" (UID: \"25f59614-ca2f-4f2d-ac69-699714a2f4c5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:23.234681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:23.234593    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:23.515537 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:23.515436    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"]
Nov 03 16:14:23.518887 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:14:23.518841    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f59614_ca2f_4f2d_ac69_699714a2f4c5.slice/crio-d14b30206a18f37fca5ac54a90a8f9d42f74d63a48fcc1474c379612358c9816 WatchSource:0}: Error finding container d14b30206a18f37fca5ac54a90a8f9d42f74d63a48fcc1474c379612358c9816: Status 404 returned error can't find the container with id d14b30206a18f37fca5ac54a90a8f9d42f74d63a48fcc1474c379612358c9816
Nov 03 16:14:24.034957 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:24.034919    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs" event={"ID":"25f59614-ca2f-4f2d-ac69-699714a2f4c5","Type":"ContainerStarted","Data":"7b157a6939381ff50d3ca20895bdc13a691f3d4cc03cc02797c943f99a285479"}
Nov 03 16:14:24.035451 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:24.035002    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs" event={"ID":"25f59614-ca2f-4f2d-ac69-699714a2f4c5","Type":"ContainerStarted","Data":"d14b30206a18f37fca5ac54a90a8f9d42f74d63a48fcc1474c379612358c9816"}
Nov 03 16:14:24.035451 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:24.035133    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:24.056539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:24.056483    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs" podStartSLOduration=1.626363438 podStartE2EDuration="2.056469624s" totalImagesPullingTime="430.106186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:24.055173559 +0000 UTC m=+725.568555336" watchObservedRunningTime="2026-11-03 16:14:24.056469624 +0000 UTC m=+725.569851399"
Nov 03 16:14:32.566131 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:14:32.566091    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:14:55.042286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.042247    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cljjs"
Nov 03 16:14:55.091583 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.091532    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"]
Nov 03 16:14:55.091843 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.091812    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck" podUID="44580bb4-9d97-4835-8175-103a37b4a2ae" containerName="manager" containerID="cri-o://ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101" gracePeriod=30
Nov 03 16:14:55.339285 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.339254    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:14:55.482202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.482160    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2\" (UniqueName: \"kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2\") pod \"44580bb4-9d97-4835-8175-103a37b4a2ae\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") "
Nov 03 16:14:55.482379 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.482209    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert\" (UniqueName: \"kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert\") pod \"44580bb4-9d97-4835-8175-103a37b4a2ae\" (UID: \"44580bb4-9d97-4835-8175-103a37b4a2ae\") "
Nov 03 16:14:55.484554 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.484521    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert" pod "44580bb4-9d97-4835-8175-103a37b4a2ae" (UID: "44580bb4-9d97-4835-8175-103a37b4a2ae"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:14:55.484668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.484593    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2" pod "44580bb4-9d97-4835-8175-103a37b4a2ae" (UID: "44580bb4-9d97-4835-8175-103a37b4a2ae"). InnerVolumeSpecName "kube-api-access-fzmn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:14:55.583907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.583825    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmn2\" (UniqueName: \"kubernetes.io/projected/44580bb4-9d97-4835-8175-103a37b4a2ae-kube-api-access-fzmn2\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:14:55.583907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:55.583858    4964 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44580bb4-9d97-4835-8175-103a37b4a2ae-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:14:56.189693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.189658    4964 generic.go:356] "Generic (PLEG): container finished" podID="44580bb4-9d97-4835-8175-103a37b4a2ae" containerID="ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101" exitCode=0
Nov 03 16:14:56.190125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.189710    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"
Nov 03 16:14:56.190125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.189738    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck" event={"ID":"44580bb4-9d97-4835-8175-103a37b4a2ae","Type":"ContainerDied","Data":"ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101"}
Nov 03 16:14:56.190125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.189792    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck" event={"ID":"44580bb4-9d97-4835-8175-103a37b4a2ae","Type":"ContainerDied","Data":"e0183ee36fea5f5793e24c79a4126cf4a9e574b997c853ee4813e8a7f5db6f01"}
Nov 03 16:14:56.190125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.189818    4964 scope.go:122] "RemoveContainer" containerID="ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101"
Nov 03 16:14:56.199271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.199245    4964 scope.go:122] "RemoveContainer" containerID="ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101"
Nov 03 16:14:56.199615 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:14:56.199593    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101\": container with ID starting with ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101 not found: ID does not exist" containerID="ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101"
Nov 03 16:14:56.199691 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.199624    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101"} err="failed to get container status \"ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101\": rpc error: code = NotFound desc = could not find container \"ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101\": container with ID starting with ba9b9bfae6a8908120170df6fc1771aebe344cf017ef03d0e3cd2d81797c2101 not found: ID does not exist"
Nov 03 16:14:56.213311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.213285    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"]
Nov 03 16:14:56.215869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.215847    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck"]
Nov 03 16:14:56.607477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:14:56.607401    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44580bb4-9d97-4835-8175-103a37b4a2ae" path="/var/lib/kubelet/pods/44580bb4-9d97-4835-8175-103a37b4a2ae/volumes"
Nov 03 16:15:02.565747 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:15:02.565662    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:15:14.627730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:14.627695    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:15:16.694590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:16.694549    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:15:18.930480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:18.930446    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:15:21.993395 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:21.993342    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:15:26.405377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:26.405336    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:15:32.566550 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:15:32.566510    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:15:36.071320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.071286    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"]
Nov 03 16:15:36.071920 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.071899    4964 cpu_manager.go:414] "containerMap: removing container" podUID="44580bb4-9d97-4835-8175-103a37b4a2ae" containerName="manager"
Nov 03 16:15:36.072007 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.071922    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44580bb4-9d97-4835-8175-103a37b4a2ae" containerName="manager"
Nov 03 16:15:36.072107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.072046    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="44580bb4-9d97-4835-8175-103a37b4a2ae" containerName="manager"
Nov 03 16:15:36.075202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.075180    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.079641 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.079611    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s22x6\""
Nov 03 16:15:36.091973 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.091942    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"]
Nov 03 16:15:36.118437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118370    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118434    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118512    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118548    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118579    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118640    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118686    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118713    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nmt\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.118748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.118731    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.219662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.219662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219674    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nmt\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.219919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219700    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.219919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219765    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.219919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219795    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219915    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.219974    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220006    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220064    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220109    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220172    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220322    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220526 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220496    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.220920 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.220896    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.222245 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.222220    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.222349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.222267    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.233017 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.232989    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.233166 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.232999    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nmt\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7z9tg\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.388468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.388359    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:36.673560 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.673524    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"]
Nov 03 16:15:36.677130 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:15:36.677102    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5413eb_9a57_4209_a52f_b0c3d0408365.slice/crio-bbb082137e5dca5d101b1c26d9269cb2fb8894fed37869b9556ec4787976185f WatchSource:0}: Error finding container bbb082137e5dca5d101b1c26d9269cb2fb8894fed37869b9556ec4787976185f: Status 404 returned error can't find the container with id bbb082137e5dca5d101b1c26d9269cb2fb8894fed37869b9556ec4787976185f
Nov 03 16:15:36.679429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.679371    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:36.679521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.679475    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:36.679567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:36.679519    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:37.397055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:37.397014    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" event={"ID":"5a5413eb-9a57-4209-a52f-b0c3d0408365","Type":"ContainerStarted","Data":"060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb"}
Nov 03 16:15:37.397516 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:37.397068    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" event={"ID":"5a5413eb-9a57-4209-a52f-b0c3d0408365","Type":"ContainerStarted","Data":"bbb082137e5dca5d101b1c26d9269cb2fb8894fed37869b9556ec4787976185f"}
Nov 03 16:15:37.426412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:37.423674    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podStartSLOduration=1.4236512559999999 podStartE2EDuration="1.423651256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:37.421945727 +0000 UTC m=+798.935327507" watchObservedRunningTime="2026-11-03 16:15:37.423651256 +0000 UTC m=+798.937033034"
Nov 03 16:15:38.389575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:38.389538    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:38.391150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:38.391121    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:15:38.391291 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:38.391173    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 16:15:39.388971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:39.388936    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:15:39.389431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:39.388991    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 16:15:40.389702 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:40.389669    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:15:40.390191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:40.389724    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 16:15:41.389019 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:41.388985    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:15:41.389214 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:41.389039    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 16:15:42.389724 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:42.389692    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:15:42.390121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:42.389748    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 16:15:43.389494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:43.389453    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 16:15:43.389674 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:43.389525    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 16:15:45.010935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:45.010902    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:45.011495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:45.011473    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:45.012487 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:45.012467    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:15:51.104502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.104468    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"]
Nov 03 16:15:51.109468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.109448    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.129155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.129123    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"]
Nov 03 16:15:51.165624 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165587    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.165787 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165644    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.165787 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165697    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86xx\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.165859 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165793    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.165859 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165826    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.165859 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165845    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.165953 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.165905    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.166028 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.166014    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.166076 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.166035    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.178539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.178505    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"]
Nov 03 16:15:51.182185 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.182167    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.207872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.207838    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"]
Nov 03 16:15:51.266591 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266556    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.266591 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266602    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.266831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266630    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.266831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266749    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.266831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266796    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.266976 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266865    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.266976 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266921    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.266978    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267026    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267052    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267059    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267121    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267139    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tbt\" (UniqueName: \"kubernetes.io/projected/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-kube-api-access-r7tbt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267168    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267199    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267226    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267242    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86xx\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267277    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267296    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267300    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.267524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267339    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267821 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267543    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.267821 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.267730    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.269077 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.269059    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.269749 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.269732    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.280404 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.280371    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.280548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.280480    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86xx\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6bg5v\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.368461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368367    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368419    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368475    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368502    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368542    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368572    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368603    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tbt\" (UniqueName: \"kubernetes.io/projected/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-kube-api-access-r7tbt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.368685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368660    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.369049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368906    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.369049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.368995    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.369049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.369029    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.369278 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.369258    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.369318 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.369294    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.370801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.370776    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.371026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.371005    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.382991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.382960    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.383320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.383300    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tbt\" (UniqueName: \"kubernetes.io/projected/e2dc90c6-0535-4ae9-bb93-5c6ee5f68345-kube-api-access-r7tbt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zphl8\" (UID: \"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.423198 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.423163    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:51.495261 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.495234    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:51.705167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.705139    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"]
Nov 03 16:15:51.707060 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:15:51.707032    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1c9f81_73b7_403d_88f9_8f6c0fa21fcb.slice/crio-fe255d47c88855a93bfddf40c3114bd5f3e27f83f0655ccdfd17a5bf70c2bbf5 WatchSource:0}: Error finding container fe255d47c88855a93bfddf40c3114bd5f3e27f83f0655ccdfd17a5bf70c2bbf5: Status 404 returned error can't find the container with id fe255d47c88855a93bfddf40c3114bd5f3e27f83f0655ccdfd17a5bf70c2bbf5
Nov 03 16:15:51.709097 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.709066    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:51.709175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.709129    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:51.709175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.709162    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:51.778852 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.778819    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"]
Nov 03 16:15:51.780595 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:15:51.780566    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dc90c6_0535_4ae9_bb93_5c6ee5f68345.slice/crio-8d12f320bd3ceec272d7779143e16a4c33c777e67be107f2be9d244eab068f12 WatchSource:0}: Error finding container 8d12f320bd3ceec272d7779143e16a4c33c777e67be107f2be9d244eab068f12: Status 404 returned error can't find the container with id 8d12f320bd3ceec272d7779143e16a4c33c777e67be107f2be9d244eab068f12
Nov 03 16:15:51.783357 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.783319    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:51.783490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.783418    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:51.783490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:51.783453    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:52.500439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:52.499968    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" event={"ID":"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345","Type":"ContainerStarted","Data":"f2113b8a6676e510323f1ba07a27a740aa9ad50a86eda15c6e0e2c4797d83857"}
Nov 03 16:15:52.500439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:52.500018    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" event={"ID":"e2dc90c6-0535-4ae9-bb93-5c6ee5f68345","Type":"ContainerStarted","Data":"8d12f320bd3ceec272d7779143e16a4c33c777e67be107f2be9d244eab068f12"}
Nov 03 16:15:52.501662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:52.501637    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" event={"ID":"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb","Type":"ContainerStarted","Data":"50d56c120960c4a36ea361d3a5bb24e5055124e838684f4ac27df5633c313a84"}
Nov 03 16:15:52.501769 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:52.501686    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" event={"ID":"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb","Type":"ContainerStarted","Data":"fe255d47c88855a93bfddf40c3114bd5f3e27f83f0655ccdfd17a5bf70c2bbf5"}
Nov 03 16:15:52.552414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:52.546798    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podStartSLOduration=1.54677731 podStartE2EDuration="1.54677731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:52.539932801 +0000 UTC m=+814.053314578" watchObservedRunningTime="2026-11-03 16:15:52.54677731 +0000 UTC m=+814.060159086"
Nov 03 16:15:52.570693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:52.570637    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podStartSLOduration=1.570622993 podStartE2EDuration="1.570622993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:52.568775723 +0000 UTC m=+814.082157501" watchObservedRunningTime="2026-11-03 16:15:52.570622993 +0000 UTC m=+814.084004768"
Nov 03 16:15:53.424411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:53.423793    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:15:53.442413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:53.440544    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:15:53.442413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:53.440616    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 16:15:53.496277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:53.496231    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:15:53.497424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:53.497399    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:15:53.497495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:53.497450    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 03 16:15:54.427035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:54.426998    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:15:54.427714 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:54.427665    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 16:15:54.495718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:54.495687    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:15:54.495958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:54.495884    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 03 16:15:55.423887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:55.423844    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:15:55.424147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:55.424116    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 16:15:55.496076 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:55.496034    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:15:55.496495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:55.496095    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 03 16:15:56.423885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:56.423842    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:15:56.424220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:56.424186    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 16:15:56.496034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:56.495994    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:15:56.496204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:56.496045    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 03 16:15:57.426413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:57.425826    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:15:57.426413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:57.425891    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 16:15:57.496559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:57.496525    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:15:57.496947 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:57.496581    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 03 16:15:58.424346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:58.424299    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 16:15:58.424537 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:58.424374    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 16:15:58.495744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:58.495703    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 03 16:15:58.495901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:15:58.495772    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 03 16:16:00.122107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:00.122065    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:16:00.122964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:00.122938    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:16:00.123911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:00.123884    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8"
Nov 03 16:16:00.129605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:00.129557    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:16:00.129988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:00.129956    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:16:00.130919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:00.130900    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:16:02.568256 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:16:02.568214    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:16:32.567419 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:16:32.567308    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:16:46.490240 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.490201    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"]
Nov 03 16:16:46.494186 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.494155    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.503508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.503474    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"]
Nov 03 16:16:46.509703 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.509669    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-69cedd-dockercfg-j59vm\""
Nov 03 16:16:46.545934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.545900    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cn4h\" (UniqueName: \"kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h\") pod \"payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.545934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.545942    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.574108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.574070    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-69cedd\""
Nov 03 16:16:46.646500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.646462    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cn4h\" (UniqueName: \"kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h\") pod \"payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.646500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.646500    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.647443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.647415    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.659638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.659602    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cn4h\" (UniqueName: \"kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h\") pod \"payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:46.814123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:46.814034    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:47.121570 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:47.121541    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"]
Nov 03 16:16:47.124143 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:16:47.124110    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21428f7_a0cc_4487_81a1_8a77a969eb50.slice/crio-4dac312c7f3d3d395357fd1058a5643f75db6cb64ce0587a8e8e0c8a3f0f9a52 WatchSource:0}: Error finding container 4dac312c7f3d3d395357fd1058a5643f75db6cb64ce0587a8e8e0c8a3f0f9a52: Status 404 returned error can't find the container with id 4dac312c7f3d3d395357fd1058a5643f75db6cb64ce0587a8e8e0c8a3f0f9a52
Nov 03 16:16:47.127188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:47.127110    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:16:47.406094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:47.406058    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:16:47.834497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:47.834409    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" event={"ID":"e21428f7-a0cc-4487-81a1-8a77a969eb50","Type":"ContainerStarted","Data":"4dac312c7f3d3d395357fd1058a5643f75db6cb64ce0587a8e8e0c8a3f0f9a52"}
Nov 03 16:16:48.257086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.257051    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"]
Nov 03 16:16:48.261084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.261055    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.268471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.268437    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.268630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.268486    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfm4p\" (UniqueName: \"kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p\") pod \"payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.278300 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.278271    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"]
Nov 03 16:16:48.287669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.287639    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-afb990\""
Nov 03 16:16:48.288143 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.288122    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-afb990-dockercfg-bssgc\""
Nov 03 16:16:48.318583 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.318537    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"]
Nov 03 16:16:48.341158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.341119    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"]
Nov 03 16:16:48.341354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.341232    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.369501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.369460    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-afb990-54654745-22rvk\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.369700 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.369534    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.369700 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.369628    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfm4p\" (UniqueName: \"kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p\") pod \"payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.369825 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.369722    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk28t\" (UniqueName: \"kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t\") pod \"payload-processing-e2e-worker-w6-afb990-54654745-22rvk\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.370507 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.370483    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.397345 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.397271    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfm4p\" (UniqueName: \"kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p\") pod \"payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.471131 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.471088    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk28t\" (UniqueName: \"kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t\") pod \"payload-processing-e2e-worker-w6-afb990-54654745-22rvk\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.471337 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.471191    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-afb990-54654745-22rvk\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.472014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.471991    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-afb990-54654745-22rvk\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.484966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.484891    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk28t\" (UniqueName: \"kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t\") pod \"payload-processing-e2e-worker-w6-afb990-54654745-22rvk\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.578813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.578524    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:48.662577 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.662539    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:48.886592 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.886556    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"]
Nov 03 16:16:48.978341 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:48.978304    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"]
Nov 03 16:16:48.987630 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:16:48.987593    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ba2d14_53e6_4bfb_b8fe_0d057eb41768.slice/crio-cc3a6097ea9490cf979f221a225920c011c782a1d2108cd5b1e3776466e12599 WatchSource:0}: Error finding container cc3a6097ea9490cf979f221a225920c011c782a1d2108cd5b1e3776466e12599: Status 404 returned error can't find the container with id cc3a6097ea9490cf979f221a225920c011c782a1d2108cd5b1e3776466e12599
Nov 03 16:16:49.303454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.303407    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"]
Nov 03 16:16:49.315054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.314561    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.320049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.320001    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"]
Nov 03 16:16:49.364534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.364150    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d569fb\""
Nov 03 16:16:49.364790 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.364771    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d569fb-dockercfg-ghdkc\""
Nov 03 16:16:49.391426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.386948    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.391426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.386996    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4f8f\" (UniqueName: \"kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f\") pod \"payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.489573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.488204    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.489573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.488267    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4f8f\" (UniqueName: \"kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f\") pod \"payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.489573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.489366    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.508497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.508421    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4f8f\" (UniqueName: \"kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f\") pod \"payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.641573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.641209    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:49.849638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.849601    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" event={"ID":"81ba2d14-53e6-4bfb-b8fe-0d057eb41768","Type":"ContainerStarted","Data":"cc3a6097ea9490cf979f221a225920c011c782a1d2108cd5b1e3776466e12599"}
Nov 03 16:16:49.850897 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.850865    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" event={"ID":"923d75e2-cd92-48de-9284-6e159e0a359a","Type":"ContainerStarted","Data":"2e61f7d884029ebb481dd32933f6bc8929f13516bb35e6a1ca386edfe645347e"}
Nov 03 16:16:49.978850 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:49.978820    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"]
Nov 03 16:16:49.996651 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:16:49.992546    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e31b7d_fda1_45b9_84b2_4295406c3357.slice/crio-92ba80ca71de92f32fb1e7b46951b6a696af2f36a5b2c573a89ed463dcc2db86 WatchSource:0}: Error finding container 92ba80ca71de92f32fb1e7b46951b6a696af2f36a5b2c573a89ed463dcc2db86: Status 404 returned error can't find the container with id 92ba80ca71de92f32fb1e7b46951b6a696af2f36a5b2c573a89ed463dcc2db86
Nov 03 16:16:50.857795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.857700    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" event={"ID":"e21428f7-a0cc-4487-81a1-8a77a969eb50","Type":"ContainerStarted","Data":"703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6"}
Nov 03 16:16:50.857795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.857782    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:16:50.860028 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.859990    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" event={"ID":"81ba2d14-53e6-4bfb-b8fe-0d057eb41768","Type":"ContainerStarted","Data":"abfbe3daaf93bd704c692b1768836f2ee684de5f103c8318e84d048e0d90dd92"}
Nov 03 16:16:50.860252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.860061    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:16:50.863890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.863862    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" event={"ID":"923d75e2-cd92-48de-9284-6e159e0a359a","Type":"ContainerStarted","Data":"32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933"}
Nov 03 16:16:50.864028 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.863951    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:16:50.869203 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.869169    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" event={"ID":"44e31b7d-fda1-45b9-84b2-4295406c3357","Type":"ContainerStarted","Data":"d5d08eba354561e5ecf2404158cedafbf3daeed4b919b1cbe093f8dd2c9f49a5"}
Nov 03 16:16:50.869371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.869226    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" event={"ID":"44e31b7d-fda1-45b9-84b2-4295406c3357","Type":"ContainerStarted","Data":"92ba80ca71de92f32fb1e7b46951b6a696af2f36a5b2c573a89ed463dcc2db86"}
Nov 03 16:16:50.869371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.869325    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:16:50.882931 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.882877    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" podStartSLOduration=2.17349522 podStartE2EDuration="4.882862684s" totalImagesPullingTime="2.709367464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.882015343 +0000 UTC m=+872.395397119" watchObservedRunningTime="2026-11-03 16:16:50.882862684 +0000 UTC m=+872.396244462"
Nov 03 16:16:50.909796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.909726    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" podStartSLOduration=1.244622349 podStartE2EDuration="2.909704775s" totalImagesPullingTime="1.665082426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.907302009 +0000 UTC m=+872.420683784" watchObservedRunningTime="2026-11-03 16:16:50.909704775 +0000 UTC m=+872.423086552"
Nov 03 16:16:50.934054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.933997    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" podStartSLOduration=1.933982007 podStartE2EDuration="1.933982007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.93240664 +0000 UTC m=+872.445788418" watchObservedRunningTime="2026-11-03 16:16:50.933982007 +0000 UTC m=+872.447363825"
Nov 03 16:16:50.967125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:50.967066    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" podStartSLOduration=1.364213596 podStartE2EDuration="2.967050328s" totalImagesPullingTime="1.602836732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.965843105 +0000 UTC m=+872.479224881" watchObservedRunningTime="2026-11-03 16:16:50.967050328 +0000 UTC m=+872.480432103"
Nov 03 16:16:51.027442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.027402    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"]
Nov 03 16:16:51.031982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.031951    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.039900 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.038376    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v65q5\""
Nov 03 16:16:51.041006 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.040985    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-69cedd-serving-cert\""
Nov 03 16:16:51.048417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.048270    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"]
Nov 03 16:16:51.048417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.048349    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:16:51.048756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.048724    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:16:51.095035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.095002    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 16:16:51.102820 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.102780    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4m2\" (UniqueName: \"kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.103012 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.102887    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.103012 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.102925    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.103012 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.102964    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.203484 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.203450    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.203672 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.203504    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4m2\" (UniqueName: \"kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.203672 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.203571    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.204348 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.203764    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.204557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.204521    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.206260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.206233    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.206260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.206252    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.227683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.227649    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4m2\" (UniqueName: \"kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2\") pod \"maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.234930 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.234897    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:16:51.352520 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.352479    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:51.643520 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.643448    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"]
Nov 03 16:16:51.648783 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:16:51.648749    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c02edc_4f93_43ba_9648_6fb680df7d4f.slice/crio-892a3370a5c54a3ac6a8d382c0550e031dc3bf092bb9b3f99ab914164eb67a69 WatchSource:0}: Error finding container 892a3370a5c54a3ac6a8d382c0550e031dc3bf092bb9b3f99ab914164eb67a69: Status 404 returned error can't find the container with id 892a3370a5c54a3ac6a8d382c0550e031dc3bf092bb9b3f99ab914164eb67a69
Nov 03 16:16:51.876168 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:51.876128    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68" event={"ID":"41c02edc-4f93-43ba-9648-6fb680df7d4f","Type":"ContainerStarted","Data":"892a3370a5c54a3ac6a8d382c0550e031dc3bf092bb9b3f99ab914164eb67a69"}
Nov 03 16:16:54.914580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:54.914537    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68" event={"ID":"41c02edc-4f93-43ba-9648-6fb680df7d4f","Type":"ContainerStarted","Data":"6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4"}
Nov 03 16:16:54.915303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:54.915277    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:16:54.937940 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:54.937892    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68" podStartSLOduration=2.312479716 podStartE2EDuration="4.937880055s" totalImagesPullingTime="2.625400339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:54.93527318 +0000 UTC m=+876.448654955" watchObservedRunningTime="2026-11-03 16:16:54.937880055 +0000 UTC m=+876.451261859"
Nov 03 16:16:56.000354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:16:56.000315    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:17:00.927502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:00.927470    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:17:01.879664 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:01.879631    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:17:01.879850 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:01.879706    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:17:01.879913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:01.879881    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:17:01.879972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:01.879957    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:17:02.570701 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:17:02.570575    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:17:04.679244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.679206    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"]
Nov 03 16:17:04.685286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.685253    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.695890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.695861    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"default-dockercfg-55z92\""
Nov 03 16:17:04.698998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.698962    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"]
Nov 03 16:17:04.755640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.755584    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"simulator-w2-b132e3-kserve-self-signed-certs\""
Nov 03 16:17:04.757987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.757959    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.758136 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.758114    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.758260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.758230    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.758304 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.758289    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9zmj\" (UniqueName: \"kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.758350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.758328    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.758453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.758439    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.782709 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.782666    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"kube-root-ca.crt\""
Nov 03 16:17:04.782901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.782743    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"openshift-service-ca.crt\""
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.859666    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.859731    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.859783    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.859878    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.859933    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.859973    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9zmj\" (UniqueName: \"kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.860820    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.861438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.861084    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.862828 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.862802    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.865884 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.865201    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.866581 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.866549    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:04.876496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:04.876460    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9zmj\" (UniqueName: \"kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj\") pod \"simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") " pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:05.010424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:05.010313    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:05.314737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:05.314708    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"]
Nov 03 16:17:05.317472 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:05.317381    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca6a6bd4_9ace_4edd_9ded_a0eb4ce04469.slice/crio-0ec2803a2de64e9cef32b83947eb61fd27f7f264d7c5826640864b6f867a0787 WatchSource:0}: Error finding container 0ec2803a2de64e9cef32b83947eb61fd27f7f264d7c5826640864b6f867a0787: Status 404 returned error can't find the container with id 0ec2803a2de64e9cef32b83947eb61fd27f7f264d7c5826640864b6f867a0787
Nov 03 16:17:05.986315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:05.986275    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" event={"ID":"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469","Type":"ContainerStarted","Data":"0ec2803a2de64e9cef32b83947eb61fd27f7f264d7c5826640864b6f867a0787"}
Nov 03 16:17:06.993466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:06.993423    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" event={"ID":"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469","Type":"ContainerStarted","Data":"12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2"}
Nov 03 16:17:07.011428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:07.011374    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:07.014809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:07.014757    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podStartSLOduration=1.731162237 podStartE2EDuration="3.014741437s" totalImagesPullingTime="1.2835792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:07.01322779 +0000 UTC m=+888.526609576" watchObservedRunningTime="2026-11-03 16:17:07.014741437 +0000 UTC m=+888.528123213"
Nov 03 16:17:07.384492 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:07.384405    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:17:10.007957 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.007912    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"]
Nov 03 16:17:10.012760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.012727    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.026482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.026431    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"]
Nov 03 16:17:10.026968 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.026943    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"simulator-w3-d7196a-kserve-self-signed-certs\""
Nov 03 16:17:10.027118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.027104    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"default-dockercfg-wmpn8\""
Nov 03 16:17:10.088737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.088685    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"kube-root-ca.crt\""
Nov 03 16:17:10.088737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.088710    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"openshift-service-ca.crt\""
Nov 03 16:17:10.128878 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.128846    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.129073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.128898    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.129073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.129007    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.129162 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.129129    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.129206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.129189    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.129267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.129250    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzfn\" (UniqueName: \"kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230195    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230244    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzfn\" (UniqueName: \"kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230324    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230367    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230426    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230490    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230703    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.230984 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.230936    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.231314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.231292    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.233575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.233552    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.234511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.234483    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.247433 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.247374    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzfn\" (UniqueName: \"kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn\") pod \"simulator-w3-d7196a-kserve-5bc994779b-bjgj6\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") " pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.335454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.335350    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:10.633582 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:10.633549    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"]
Nov 03 16:17:10.635984 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:10.635951    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c4e6063_5b41_4e2d_a256_e009c9d5adcf.slice/crio-71661ee4bfc8f2c39d4bbf44a5cab6e697da3b79ce5ad125da8afd969b92b5d6 WatchSource:0}: Error finding container 71661ee4bfc8f2c39d4bbf44a5cab6e697da3b79ce5ad125da8afd969b92b5d6: Status 404 returned error can't find the container with id 71661ee4bfc8f2c39d4bbf44a5cab6e697da3b79ce5ad125da8afd969b92b5d6
Nov 03 16:17:11.025443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:11.025400    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" event={"ID":"5c4e6063-5b41-4e2d-a256-e009c9d5adcf","Type":"ContainerStarted","Data":"4d02e42ded70611f545a6b57a540ee20bd58ec809e296cf597d4f87422e150bd"}
Nov 03 16:17:11.025443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:11.025447    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" event={"ID":"5c4e6063-5b41-4e2d-a256-e009c9d5adcf","Type":"ContainerStarted","Data":"71661ee4bfc8f2c39d4bbf44a5cab6e697da3b79ce5ad125da8afd969b92b5d6"}
Nov 03 16:17:11.336224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:11.336123    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:12.631655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.631575    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podStartSLOduration=3.631551601 podStartE2EDuration="3.631551601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:11.050655288 +0000 UTC m=+892.564037064" watchObservedRunningTime="2026-11-03 16:17:12.631551601 +0000 UTC m=+894.144933377"
Nov 03 16:17:12.633358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.633321    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"]
Nov 03 16:17:12.638181 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.638154    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.647710 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.647683    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"openshift-service-ca.crt\""
Nov 03 16:17:12.648369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.648347    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"kube-root-ca.crt\""
Nov 03 16:17:12.648590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.648564    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"default-dockercfg-vkbzw\""
Nov 03 16:17:12.649140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.648459    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"simulator-w5-69cedd-kserve-self-signed-certs\""
Nov 03 16:17:12.651282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.651258    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"]
Nov 03 16:17:12.754473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.754415    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.754684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.754481    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blx2\" (UniqueName: \"kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.754684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.754615    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.754684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.754677    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.754844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.754702    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.754844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.754746    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.855915 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.855878    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blx2\" (UniqueName: \"kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.855929    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.855972    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.855992    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856263 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.856130    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856363 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.856341    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.856499    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.856567    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.856788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.856763    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.858572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.858544    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.858776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.858760    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.870849 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.870815    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blx2\" (UniqueName: \"kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2\") pod \"simulator-w5-69cedd-kserve-556f678db7-fjfm7\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") " pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:12.953917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:12.953878    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:13.237344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:13.237012    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"]
Nov 03 16:17:13.239985 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:13.239945    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf81471_76cc_4465_8a94_bfdd83b764cb.slice/crio-ed916fab157eb34a22b33bbabed3337591e8f04eea5a7667c31ee573a4cc9c18 WatchSource:0}: Error finding container ed916fab157eb34a22b33bbabed3337591e8f04eea5a7667c31ee573a4cc9c18: Status 404 returned error can't find the container with id ed916fab157eb34a22b33bbabed3337591e8f04eea5a7667c31ee573a4cc9c18
Nov 03 16:17:14.061885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:14.061840    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" event={"ID":"ecf81471-76cc-4465-8a94-bfdd83b764cb","Type":"ContainerStarted","Data":"21488714470cf06a5d0b4c54dec18245c5c871ca982ec5358b82fef7de96c02e"}
Nov 03 16:17:14.062299 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:14.061898    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" event={"ID":"ecf81471-76cc-4465-8a94-bfdd83b764cb","Type":"ContainerStarted","Data":"ed916fab157eb34a22b33bbabed3337591e8f04eea5a7667c31ee573a4cc9c18"}
Nov 03 16:17:14.092610 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:14.090456    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podStartSLOduration=2.090435347 podStartE2EDuration="2.090435347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:14.089543778 +0000 UTC m=+895.602925560" watchObservedRunningTime="2026-11-03 16:17:14.090435347 +0000 UTC m=+895.603817126"
Nov 03 16:17:14.954398 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:14.954343    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:15.011454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:15.011413    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:15.030205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:15.030171    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:15.084375 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:15.084341    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:17:17.609238 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:17.609198    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:17:18.702720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:18.702688    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:17:18.704004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:18.703980    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:17:20.267366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.267332    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"]
Nov 03 16:17:20.272338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.272312    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.281977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.281948    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"simulator-w4-4a19cd-kserve-self-signed-certs\""
Nov 03 16:17:20.282114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.282097    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"default-dockercfg-4mpx9\""
Nov 03 16:17:20.282185 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.282177    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"kube-root-ca.crt\""
Nov 03 16:17:20.282267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.282254    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"openshift-service-ca.crt\""
Nov 03 16:17:20.284820 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.284791    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"]
Nov 03 16:17:20.335553 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.335512    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:20.349548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.349506    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:20.436879 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.436835    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.437079 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.436933    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.437079 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.436980    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.437079 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.437023    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.437079 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.437075    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86jk\" (UniqueName: \"kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.437312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.437095    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.538742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538343    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.538742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538470    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.538742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538503    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.538742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538534    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.538742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538565    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86jk\" (UniqueName: \"kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.538742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538587    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.539175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538870    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.539175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.538982    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.539175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.539142    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.541105 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.541075    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.541733 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.541710    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.553834 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.553804    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86jk\" (UniqueName: \"kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk\") pod \"simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") " pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.587106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.587073    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:20.883523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:20.883495    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"]
Nov 03 16:17:20.884980 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:20.884953    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dacc118_e0f5_4abb_8caa_9547cb242155.slice/crio-e13cc67a395c89c55414a7968b18cca9301f3228229e2f604a29ac8872c3eb4e WatchSource:0}: Error finding container e13cc67a395c89c55414a7968b18cca9301f3228229e2f604a29ac8872c3eb4e: Status 404 returned error can't find the container with id e13cc67a395c89c55414a7968b18cca9301f3228229e2f604a29ac8872c3eb4e
Nov 03 16:17:21.112452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:21.112405    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" event={"ID":"5dacc118-e0f5-4abb-8caa-9547cb242155","Type":"ContainerStarted","Data":"1106fe7f3e3c80981cd955440b87cb32806a0c316e40e38485b11d27e20ddc29"}
Nov 03 16:17:21.112723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:21.112471    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" event={"ID":"5dacc118-e0f5-4abb-8caa-9547cb242155","Type":"ContainerStarted","Data":"e13cc67a395c89c55414a7968b18cca9301f3228229e2f604a29ac8872c3eb4e"}
Nov 03 16:17:21.129022 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:21.128991    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:17:21.136012 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:21.135893    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podStartSLOduration=1.135878925 podStartE2EDuration="1.135878925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:21.134225378 +0000 UTC m=+902.647607155" watchObservedRunningTime="2026-11-03 16:17:21.135878925 +0000 UTC m=+902.649260704"
Nov 03 16:17:21.587676 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:21.587638    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:22.954433 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:22.954377    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:22.967811 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:22.967782    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:23.145109 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:23.145079    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:17:27.645980 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.645929    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"]
Nov 03 16:17:27.656219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.656189    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.660703 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.660677    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"simulator-w1-c09b17-kserve-self-signed-certs\""
Nov 03 16:17:27.661242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.661211    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"]
Nov 03 16:17:27.664653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.664632    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"kube-root-ca.crt\""
Nov 03 16:17:27.664867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.664844    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"openshift-service-ca.crt\""
Nov 03 16:17:27.702121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.702089    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"default-dockercfg-xbvrx\""
Nov 03 16:17:27.819754 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.819710    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.819950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.819760    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.819950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.819837    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.819950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.819921    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl644\" (UniqueName: \"kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.820111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.819970    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.820111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.819994    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.920972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.920929    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.920972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.920970    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.921226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921041    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.921226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921073    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.921226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921102    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.921226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921136    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl644\" (UniqueName: \"kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.921527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921467    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.921587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921550    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.922013 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.921725    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.923575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.923542    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.923927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.923900    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.934712 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.934682    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl644\" (UniqueName: \"kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644\") pod \"simulator-w1-c09b17-kserve-6dc845f788-fk9sk\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") " pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:27.974756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:27.974717    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:28.260058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:28.260031    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"]
Nov 03 16:17:28.262486 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:28.262457    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89127b8_55cb_4686_9d16_7009d7cca6be.slice/crio-1a215028ea7e9e8f150c41161c8492c2fd89d1473f1c7fd6cec527e2191868a2 WatchSource:0}: Error finding container 1a215028ea7e9e8f150c41161c8492c2fd89d1473f1c7fd6cec527e2191868a2: Status 404 returned error can't find the container with id 1a215028ea7e9e8f150c41161c8492c2fd89d1473f1c7fd6cec527e2191868a2
Nov 03 16:17:29.185611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:29.185562    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" event={"ID":"b89127b8-55cb-4686-9d16-7009d7cca6be","Type":"ContainerStarted","Data":"40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef"}
Nov 03 16:17:29.186031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:29.185620    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" event={"ID":"b89127b8-55cb-4686-9d16-7009d7cca6be","Type":"ContainerStarted","Data":"1a215028ea7e9e8f150c41161c8492c2fd89d1473f1c7fd6cec527e2191868a2"}
Nov 03 16:17:29.213780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:29.213707    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podStartSLOduration=2.21368559 podStartE2EDuration="2.21368559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:29.210125904 +0000 UTC m=+910.723507682" watchObservedRunningTime="2026-11-03 16:17:29.21368559 +0000 UTC m=+910.727067373"
Nov 03 16:17:29.975567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:29.975520    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:30.587778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:30.587743    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:30.613748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:30.613716    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:31.063032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:31.062993    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:17:31.209572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:31.209527    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:17:32.566882 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:17:32.566851    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:17:37.563514 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.563473    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"]
Nov 03 16:17:37.567693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.567665    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.578283 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.578256    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"premium-w3-d7196a-kserve-self-signed-certs\""
Nov 03 16:17:37.580584 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.580554    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"]
Nov 03 16:17:37.618987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.618935    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l46x\" (UniqueName: \"kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.619190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.619043    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.619190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.619133    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.619300 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.619260    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.619362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.619348    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.619431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.619420    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.720247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.720200    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.721043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.721017    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.721253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.721237    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.721375 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.721360    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.721550 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.721533    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l46x\" (UniqueName: \"kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.721668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.721656    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.722134 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.722114    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.722505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.722484    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.722844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.722826    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.733135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.733100    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.750346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.750309    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l46x\" (UniqueName: \"kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.755777 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.755741    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm\") pod \"premium-w3-d7196a-kserve-7cc8557f6f-k9wm2\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") " pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.884326 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.884227    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:37.976054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.975865    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:37.993837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:37.993674    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:38.176611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:38.176578    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"]
Nov 03 16:17:38.177655 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:38.177627    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaec860b_340b_4777_9b61_b525bb9c42f6.slice/crio-6f3c3d1af6a4bf0a25a74ffd92b9265f9f274097bc6d9b2db8715cdb7f620528 WatchSource:0}: Error finding container 6f3c3d1af6a4bf0a25a74ffd92b9265f9f274097bc6d9b2db8715cdb7f620528: Status 404 returned error can't find the container with id 6f3c3d1af6a4bf0a25a74ffd92b9265f9f274097bc6d9b2db8715cdb7f620528
Nov 03 16:17:38.244929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:38.244512    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" event={"ID":"caec860b-340b-4777-9b61-b525bb9c42f6","Type":"ContainerStarted","Data":"6f3c3d1af6a4bf0a25a74ffd92b9265f9f274097bc6d9b2db8715cdb7f620528"}
Nov 03 16:17:38.262140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:38.261948    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:17:39.252487 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:39.252439    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" event={"ID":"caec860b-340b-4777-9b61-b525bb9c42f6","Type":"ContainerStarted","Data":"1ca15c243b6147d4794be8440b27daf65be2a2857c4113a88f1b8fc0fd082e05"}
Nov 03 16:17:39.303688 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:39.303619    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podStartSLOduration=2.3035895809999998 podStartE2EDuration="2.303589581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:39.30151849 +0000 UTC m=+920.814900266" watchObservedRunningTime="2026-11-03 16:17:39.303589581 +0000 UTC m=+920.816971360"
Nov 03 16:17:39.885114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:39.885081    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:43.508059 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.508019    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"]
Nov 03 16:17:43.515640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.515611    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.523693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.523660    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"]
Nov 03 16:17:43.528562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.528530    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"openshift-service-ca.crt\""
Nov 03 16:17:43.528729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.528663    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"default-dockercfg-dq8hv\""
Nov 03 16:17:43.528792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.528746    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"premium-w0-d569fb-kserve-self-signed-certs\""
Nov 03 16:17:43.528856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.528838    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"kube-root-ca.crt\""
Nov 03 16:17:43.587926 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.587877    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.588123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.587935    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.588123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.588045    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.588123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.588077    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.588304 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.588173    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.588304 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.588244    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgh4\" (UniqueName: \"kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.689353 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.689311    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgh4\" (UniqueName: \"kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.689578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.689408    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.689578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.689427    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.689578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.689459    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.689578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.689477    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.689969 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.689939    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.690065 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.690001    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.690130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.690099    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.690511 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.690485    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.692061 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.692036    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.692325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.692301    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.702562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.702530    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgh4\" (UniqueName: \"kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4\") pod \"premium-w0-d569fb-kserve-5b5ff7959b-hc6f5\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") " pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:43.852080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:43.851983    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:44.140171 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:44.140139    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"]
Nov 03 16:17:44.142287 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:44.142256    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec89f4cf_6327_4b51_9511_8800454e76c9.slice/crio-4e5a4f6f4c5f6b89a34e6bc3fbac186c508baff1852bc9d9680265d4230ebe31 WatchSource:0}: Error finding container 4e5a4f6f4c5f6b89a34e6bc3fbac186c508baff1852bc9d9680265d4230ebe31: Status 404 returned error can't find the container with id 4e5a4f6f4c5f6b89a34e6bc3fbac186c508baff1852bc9d9680265d4230ebe31
Nov 03 16:17:44.282634 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:44.282595    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" event={"ID":"ec89f4cf-6327-4b51-9511-8800454e76c9","Type":"ContainerStarted","Data":"cb73a208731ebc6ce29ab4f49eb861f5837d6e6792548fae5528e170db6bdae6"}
Nov 03 16:17:44.282839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:44.282648    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" event={"ID":"ec89f4cf-6327-4b51-9511-8800454e76c9","Type":"ContainerStarted","Data":"4e5a4f6f4c5f6b89a34e6bc3fbac186c508baff1852bc9d9680265d4230ebe31"}
Nov 03 16:17:44.308150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:44.308098    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podStartSLOduration=1.308082361 podStartE2EDuration="1.308082361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:44.305585799 +0000 UTC m=+925.818967575" watchObservedRunningTime="2026-11-03 16:17:44.308082361 +0000 UTC m=+925.821464136"
Nov 03 16:17:44.852363 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:44.852330    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:45.171039 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:45.171001    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:17:47.885095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:47.885055    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:47.897942 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:47.897912    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:48.318117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:48.318087    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:17:53.852735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:53.852685    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:53.866002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:53.865974    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:54.361275 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:54.361242    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:17:56.533534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.533495    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"]
Nov 03 16:17:56.538794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.538768    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.553848 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.553289    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"premium-w1-c09b17-kserve-self-signed-certs\""
Nov 03 16:17:56.556343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.556311    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"]
Nov 03 16:17:56.635137 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.635097    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.635342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.635190    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.635342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.635227    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptn26\" (UniqueName: \"kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.635342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.635318    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.635540 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.635442    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.635540 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.635502    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.736333 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.736286    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.736557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.736367    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.736557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.736524    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.736748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.736726    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.736844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.736826    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.736907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.736857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptn26\" (UniqueName: \"kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.737179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.737150    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.737478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.737431    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.737819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.737763    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.739567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.739542    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.739756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.739736    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.749684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.749652    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptn26\" (UniqueName: \"kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26\") pod \"premium-w1-c09b17-kserve-7664bcd5fc-tm2hh\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") " pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:56.858653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:56.858568    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:17:57.144509 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:57.144441    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"]
Nov 03 16:17:57.147129 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:17:57.147088    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98c8edbf_14c3_4af7_8f15_e0b4b919b625.slice/crio-8a18fb6ed6a6208e8eb260b9c3c313f3f9aa59ed095416f08ffa010b8bd9d393 WatchSource:0}: Error finding container 8a18fb6ed6a6208e8eb260b9c3c313f3f9aa59ed095416f08ffa010b8bd9d393: Status 404 returned error can't find the container with id 8a18fb6ed6a6208e8eb260b9c3c313f3f9aa59ed095416f08ffa010b8bd9d393
Nov 03 16:17:57.369484 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:57.369441    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" event={"ID":"98c8edbf-14c3-4af7-8f15-e0b4b919b625","Type":"ContainerStarted","Data":"a32790e4514bdb37a2c593d4004f3a215edb5525978e0b62c9ce7eb3fd5849fe"}
Nov 03 16:17:57.369685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:57.369491    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" event={"ID":"98c8edbf-14c3-4af7-8f15-e0b4b919b625","Type":"ContainerStarted","Data":"8a18fb6ed6a6208e8eb260b9c3c313f3f9aa59ed095416f08ffa010b8bd9d393"}
Nov 03 16:17:57.389934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:57.389875    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podStartSLOduration=1.38985435 podStartE2EDuration="1.38985435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:57.389441226 +0000 UTC m=+938.902823004" watchObservedRunningTime="2026-11-03 16:17:57.38985435 +0000 UTC m=+938.903236128"
Nov 03 16:17:57.859146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:17:57.859101    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:18:02.296963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:02.296877    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:18:02.568036 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:18:02.567955    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:18:06.859185 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:06.859137    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:18:06.872274 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:06.872242    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:18:07.440476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:07.440445    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:18:18.092265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.092230    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"]
Nov 03 16:18:18.097808 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.097783    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.105421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.105375    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:18.106983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.106944    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:18.110472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.110437    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"]
Nov 03 16:18:18.186922 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.186888    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.187129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.186960    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvk9f\" (UniqueName: \"kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.187129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.187088    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.287927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.287889    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvk9f\" (UniqueName: \"kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.288130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.287992    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.288130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.288057    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.288885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.288857    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.290840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.290812    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.309248 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.309213    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvk9f\" (UniqueName: \"kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f\") pod \"maas-api-revoke-sub-5d64f441-5g7zm\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.415901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.415860    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:18.701988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:18.701963    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"]
Nov 03 16:18:18.704151 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:18.704125    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e9c7d3_c760_4427_9e04_5eef25842147.slice/crio-e2859e96adc6fae1db15a99d73ff2a77c8fb59d8cd5b382bd0084523ffd38096 WatchSource:0}: Error finding container e2859e96adc6fae1db15a99d73ff2a77c8fb59d8cd5b382bd0084523ffd38096: Status 404 returned error can't find the container with id e2859e96adc6fae1db15a99d73ff2a77c8fb59d8cd5b382bd0084523ffd38096
Nov 03 16:18:19.513165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:19.513127    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm" event={"ID":"d4e9c7d3-c760-4427-9e04-5eef25842147","Type":"ContainerStarted","Data":"e2859e96adc6fae1db15a99d73ff2a77c8fb59d8cd5b382bd0084523ffd38096"}
Nov 03 16:18:22.535568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:22.535532    4964 generic.go:356] "Generic (PLEG): container finished" podID="d4e9c7d3-c760-4427-9e04-5eef25842147" containerID="3e6c38794abda890dac7bc5708548b351c7e9f8cd0f38c424c30a9aa67c9a4c7" exitCode=0
Nov 03 16:18:22.536101 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:22.535607    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm" event={"ID":"d4e9c7d3-c760-4427-9e04-5eef25842147","Type":"ContainerDied","Data":"3e6c38794abda890dac7bc5708548b351c7e9f8cd0f38c424c30a9aa67c9a4c7"}
Nov 03 16:18:23.687940 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.687909    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:23.746796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.746752    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f\" (UniqueName: \"kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f\") pod \"d4e9c7d3-c760-4427-9e04-5eef25842147\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") "
Nov 03 16:18:23.746987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.746942    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca\") pod \"d4e9c7d3-c760-4427-9e04-5eef25842147\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") "
Nov 03 16:18:23.747051 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.747012    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca\") pod \"d4e9c7d3-c760-4427-9e04-5eef25842147\" (UID: \"d4e9c7d3-c760-4427-9e04-5eef25842147\") "
Nov 03 16:18:23.747438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.747405    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca" pod "d4e9c7d3-c760-4427-9e04-5eef25842147" (UID: "d4e9c7d3-c760-4427-9e04-5eef25842147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:23.750343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.750307    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca" pod "d4e9c7d3-c760-4427-9e04-5eef25842147" (UID: "d4e9c7d3-c760-4427-9e04-5eef25842147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:23.750522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.750340    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f" pod "d4e9c7d3-c760-4427-9e04-5eef25842147" (UID: "d4e9c7d3-c760-4427-9e04-5eef25842147"). InnerVolumeSpecName "kube-api-access-nvk9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:23.849046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.848945    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:23.849046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.848986    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9c7d3-c760-4427-9e04-5eef25842147-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:23.849046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:23.849001    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvk9f\" (UniqueName: \"kubernetes.io/projected/d4e9c7d3-c760-4427-9e04-5eef25842147-kube-api-access-nvk9f\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:24.522810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.522770    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"]
Nov 03 16:18:24.523344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.523324    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d4e9c7d3-c760-4427-9e04-5eef25842147" containerName="revoke-keys"
Nov 03 16:18:24.523478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.523346    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e9c7d3-c760-4427-9e04-5eef25842147" containerName="revoke-keys"
Nov 03 16:18:24.523582 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.523567    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e9c7d3-c760-4427-9e04-5eef25842147" containerName="revoke-keys"
Nov 03 16:18:24.527306 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.527283    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.532150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.532124    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d569fb-openshift-default-dockercfg-wtrd5\""
Nov 03 16:18:24.542512 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.542482    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"]
Nov 03 16:18:24.553745 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.553719    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"
Nov 03 16:18:24.553935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.553710    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm" event={"ID":"d4e9c7d3-c760-4427-9e04-5eef25842147","Type":"ContainerDied","Data":"e2859e96adc6fae1db15a99d73ff2a77c8fb59d8cd5b382bd0084523ffd38096"}
Nov 03 16:18:24.553935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.553876    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2859e96adc6fae1db15a99d73ff2a77c8fb59d8cd5b382bd0084523ffd38096"
Nov 03 16:18:24.656492 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656447    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.656671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656521    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.656671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656602    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.656671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656640    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.656853 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656837    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.656917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656868    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.656917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656907    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.657030 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.656951    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskxj\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.657083 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.657039    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.757689 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757653    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757704    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757779    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757819    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757851    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757888    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskxj\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.757946    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.758193    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.758272    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.758453    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.758495    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758764 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.758636    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.758764 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.758699    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.759035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.759007    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.760649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.760621    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.760742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.760675    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.771001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.770977    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskxj\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.771130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.770993    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:24.847069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:24.846977    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:25.083918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.083883    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:18:25.135977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.135947    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"]
Nov 03 16:18:25.138196 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:25.138165    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077b8c38_65b7_443c_afe5_8d43d20f0a40.slice/crio-ef7ac8b02fdc6b72e6319e3dc83d6f98f824e0c841e491aaf146bb186a9c74dd WatchSource:0}: Error finding container ef7ac8b02fdc6b72e6319e3dc83d6f98f824e0c841e491aaf146bb186a9c74dd: Status 404 returned error can't find the container with id ef7ac8b02fdc6b72e6319e3dc83d6f98f824e0c841e491aaf146bb186a9c74dd
Nov 03 16:18:25.140960 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.140923    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:18:25.141056 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.140995    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:18:25.141056 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.141024    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:18:25.563741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.563699    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" event={"ID":"077b8c38-65b7-443c-afe5-8d43d20f0a40","Type":"ContainerStarted","Data":"0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d"}
Nov 03 16:18:25.563954 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.563775    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" event={"ID":"077b8c38-65b7-443c-afe5-8d43d20f0a40","Type":"ContainerStarted","Data":"ef7ac8b02fdc6b72e6319e3dc83d6f98f824e0c841e491aaf146bb186a9c74dd"}
Nov 03 16:18:25.597447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.597356    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podStartSLOduration=1.597334515 podStartE2EDuration="1.597334515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:25.590844422 +0000 UTC m=+967.104226231" watchObservedRunningTime="2026-11-03 16:18:25.597334515 +0000 UTC m=+967.110716292"
Nov 03 16:18:25.847360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:25.847278    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:26.848862 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:26.848824    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 16:18:26.849265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:26.848883    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 16:18:27.847761 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:27.847729    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 16:18:27.847975 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:27.847783    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 16:18:28.848013 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:28.847976    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 16:18:28.848440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:28.848031    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 16:18:29.847933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:29.847897    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 16:18:29.848091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:29.847953    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 16:18:30.446038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.446003    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"]
Nov 03 16:18:30.450058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.450034    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.454970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.454943    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:30.455199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.455180    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:30.467533 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.467498    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"]
Nov 03 16:18:30.541950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.541912    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6xwf\" (UniqueName: \"kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.542135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.541961    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.542135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.542076    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.643111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.643074    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.643274 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.643213    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6xwf\" (UniqueName: \"kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.643274 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.643252    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.644441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.644379    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.645866 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.645837    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.657194 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.657166    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6xwf\" (UniqueName: \"kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf\") pod \"maas-api-revoke-sub-64a02195-s4dlw\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.764428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.764330    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:30.848104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.848076    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 16:18:30.848440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:30.848129    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 16:18:31.072416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.072359    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"]
Nov 03 16:18:31.599103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.599060    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw" event={"ID":"6931c64c-eeb8-4226-8546-02f9d7c69de5","Type":"ContainerStarted","Data":"0f77c7152ed0e7ef75844b11d761803756a4fe6cd629214df66c23d784c14091"}
Nov 03 16:18:31.599801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.599181    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw" event={"ID":"6931c64c-eeb8-4226-8546-02f9d7c69de5","Type":"ContainerStarted","Data":"77d0567df5cdeb894e59aed3d2cd0bed0fff700f2fa6dc249bd34a98af3fe6c7"}
Nov 03 16:18:31.634281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.634216    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw" podStartSLOduration=1.634195678 podStartE2EDuration="1.634195678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:31.629788693 +0000 UTC m=+973.143170480" watchObservedRunningTime="2026-11-03 16:18:31.634195678 +0000 UTC m=+973.147577456"
Nov 03 16:18:31.875338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.875252    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:31.876031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.876011    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:31.877753 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:31.877328    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:18:32.566504 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:18:32.566469    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:18:32.611038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:32.610991    4964 generic.go:356] "Generic (PLEG): container finished" podID="6931c64c-eeb8-4226-8546-02f9d7c69de5" containerID="0f77c7152ed0e7ef75844b11d761803756a4fe6cd629214df66c23d784c14091" exitCode=0
Nov 03 16:18:32.612580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:32.612550    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw" event={"ID":"6931c64c-eeb8-4226-8546-02f9d7c69de5","Type":"ContainerDied","Data":"0f77c7152ed0e7ef75844b11d761803756a4fe6cd629214df66c23d784c14091"}
Nov 03 16:18:33.772760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.772729    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:33.878969 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.878873    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf\" (UniqueName: \"kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf\") pod \"6931c64c-eeb8-4226-8546-02f9d7c69de5\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") "
Nov 03 16:18:33.879130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.879076    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca\") pod \"6931c64c-eeb8-4226-8546-02f9d7c69de5\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") "
Nov 03 16:18:33.879203 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.879188    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca\") pod \"6931c64c-eeb8-4226-8546-02f9d7c69de5\" (UID: \"6931c64c-eeb8-4226-8546-02f9d7c69de5\") "
Nov 03 16:18:33.879608 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.879576    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca" pod "6931c64c-eeb8-4226-8546-02f9d7c69de5" (UID: "6931c64c-eeb8-4226-8546-02f9d7c69de5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:33.881220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.881199    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf" pod "6931c64c-eeb8-4226-8546-02f9d7c69de5" (UID: "6931c64c-eeb8-4226-8546-02f9d7c69de5"). InnerVolumeSpecName "kube-api-access-c6xwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:33.881295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.881201    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca" pod "6931c64c-eeb8-4226-8546-02f9d7c69de5" (UID: "6931c64c-eeb8-4226-8546-02f9d7c69de5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:33.981115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.981075    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:33.981115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.981111    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6xwf\" (UniqueName: \"kubernetes.io/projected/6931c64c-eeb8-4226-8546-02f9d7c69de5-kube-api-access-c6xwf\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:33.981115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:33.981122    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6931c64c-eeb8-4226-8546-02f9d7c69de5-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:34.625762 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:34.625730    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"
Nov 03 16:18:34.625961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:34.625760    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw" event={"ID":"6931c64c-eeb8-4226-8546-02f9d7c69de5","Type":"ContainerDied","Data":"77d0567df5cdeb894e59aed3d2cd0bed0fff700f2fa6dc249bd34a98af3fe6c7"}
Nov 03 16:18:34.625961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:34.625816    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77d0567df5cdeb894e59aed3d2cd0bed0fff700f2fa6dc249bd34a98af3fe6c7"
Nov 03 16:18:39.213363 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.213324    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"]
Nov 03 16:18:39.213872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.213840    4964 cpu_manager.go:414] "containerMap: removing container" podUID="6931c64c-eeb8-4226-8546-02f9d7c69de5" containerName="revoke-keys"
Nov 03 16:18:39.213872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.213853    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6931c64c-eeb8-4226-8546-02f9d7c69de5" containerName="revoke-keys"
Nov 03 16:18:39.213983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.213971    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="6931c64c-eeb8-4226-8546-02f9d7c69de5" containerName="revoke-keys"
Nov 03 16:18:39.217699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.217675    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.228328 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.228047    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"]
Nov 03 16:18:39.228545 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.228518    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:39.228675 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.228612    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:39.339493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.339455    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.339719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.339649    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.339719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.339679    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkzq\" (UniqueName: \"kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.442551 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.442511    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.442551 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.442549    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkzq\" (UniqueName: \"kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.442838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.442667    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.443267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.443236    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.445172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.445146    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.457463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.457421    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkzq\" (UniqueName: \"kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq\") pod \"maas-api-revoke-sub-6144a104-rbxrh\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.535336 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.535230    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:39.821490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:39.821378    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"]
Nov 03 16:18:39.825879 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:39.825842    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9312ed9d_5b54_4292_847b_91866ccd15af.slice/crio-5dea9d328817687726db85edecc94a45f2210069b08c5b65c9bf42280d59223b WatchSource:0}: Error finding container 5dea9d328817687726db85edecc94a45f2210069b08c5b65c9bf42280d59223b: Status 404 returned error can't find the container with id 5dea9d328817687726db85edecc94a45f2210069b08c5b65c9bf42280d59223b
Nov 03 16:18:40.665559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:40.665517    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh" event={"ID":"9312ed9d-5b54-4292-847b-91866ccd15af","Type":"ContainerStarted","Data":"fd650dcfa70f38b06157cf1edceef09e7e4b2b94b001f0164d265f2eb28a3f11"}
Nov 03 16:18:40.666138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:40.665584    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh" event={"ID":"9312ed9d-5b54-4292-847b-91866ccd15af","Type":"ContainerStarted","Data":"5dea9d328817687726db85edecc94a45f2210069b08c5b65c9bf42280d59223b"}
Nov 03 16:18:40.685608 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:40.685556    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh" podStartSLOduration=1.6855429389999999 podStartE2EDuration="1.685542939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:40.68423879 +0000 UTC m=+982.197620590" watchObservedRunningTime="2026-11-03 16:18:40.685542939 +0000 UTC m=+982.198924715"
Nov 03 16:18:41.673161 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:41.673120    4964 generic.go:356] "Generic (PLEG): container finished" podID="9312ed9d-5b54-4292-847b-91866ccd15af" containerID="fd650dcfa70f38b06157cf1edceef09e7e4b2b94b001f0164d265f2eb28a3f11" exitCode=0
Nov 03 16:18:41.673690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:41.673197    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh" event={"ID":"9312ed9d-5b54-4292-847b-91866ccd15af","Type":"ContainerDied","Data":"fd650dcfa70f38b06157cf1edceef09e7e4b2b94b001f0164d265f2eb28a3f11"}
Nov 03 16:18:42.848797 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.848767    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:42.880950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.880906    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq\" (UniqueName: \"kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq\") pod \"9312ed9d-5b54-4292-847b-91866ccd15af\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") "
Nov 03 16:18:42.881123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.880977    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca\") pod \"9312ed9d-5b54-4292-847b-91866ccd15af\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") "
Nov 03 16:18:42.881186 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.881132    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca\") pod \"9312ed9d-5b54-4292-847b-91866ccd15af\" (UID: \"9312ed9d-5b54-4292-847b-91866ccd15af\") "
Nov 03 16:18:42.882565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.882524    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca" pod "9312ed9d-5b54-4292-847b-91866ccd15af" (UID: "9312ed9d-5b54-4292-847b-91866ccd15af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:42.883697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.883670    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca" pod "9312ed9d-5b54-4292-847b-91866ccd15af" (UID: "9312ed9d-5b54-4292-847b-91866ccd15af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:42.883904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.883881    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq" pod "9312ed9d-5b54-4292-847b-91866ccd15af" (UID: "9312ed9d-5b54-4292-847b-91866ccd15af"). InnerVolumeSpecName "kube-api-access-9fkzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:42.925724 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.925679    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"]
Nov 03 16:18:42.926450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.926427    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9312ed9d-5b54-4292-847b-91866ccd15af" containerName="revoke-keys"
Nov 03 16:18:42.926450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.926449    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9312ed9d-5b54-4292-847b-91866ccd15af" containerName="revoke-keys"
Nov 03 16:18:42.926659 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.926643    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9312ed9d-5b54-4292-847b-91866ccd15af" containerName="revoke-keys"
Nov 03 16:18:42.930563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.930501    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:42.940167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.940139    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"]
Nov 03 16:18:42.982968 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.982926    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:42.983150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.983132    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:42.983282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.983256    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvbl\" (UniqueName: \"kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:42.983360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.983344    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkzq\" (UniqueName: \"kubernetes.io/projected/9312ed9d-5b54-4292-847b-91866ccd15af-kube-api-access-9fkzq\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:42.983430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.983366    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:42.983430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:42.983421    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312ed9d-5b54-4292-847b-91866ccd15af-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:43.084432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.084397    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.084648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.084470    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvbl\" (UniqueName: \"kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.084648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.084498    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.085085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.085061    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.086897 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.086873    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.096580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.096560    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvbl\" (UniqueName: \"kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl\") pod \"maas-api-revoke-sub-d5be5fa6-dgckb\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.249079 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.248983    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:43.529747 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.529725    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"]
Nov 03 16:18:43.531505 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:43.531478    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27107370_b018_4f71_a6b0_e3266680fe97.slice/crio-b76b9eeee8c3e0f7c18c76ec2cbb03a52fbe065d3d50913a0edb8426f95b3ff2 WatchSource:0}: Error finding container b76b9eeee8c3e0f7c18c76ec2cbb03a52fbe065d3d50913a0edb8426f95b3ff2: Status 404 returned error can't find the container with id b76b9eeee8c3e0f7c18c76ec2cbb03a52fbe065d3d50913a0edb8426f95b3ff2
Nov 03 16:18:43.686903 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.686864    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh" event={"ID":"9312ed9d-5b54-4292-847b-91866ccd15af","Type":"ContainerDied","Data":"5dea9d328817687726db85edecc94a45f2210069b08c5b65c9bf42280d59223b"}
Nov 03 16:18:43.687087 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.686925    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dea9d328817687726db85edecc94a45f2210069b08c5b65c9bf42280d59223b"
Nov 03 16:18:43.687087 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.686880    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"
Nov 03 16:18:43.688366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:43.688326    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb" event={"ID":"27107370-b018-4f71-a6b0-e3266680fe97","Type":"ContainerStarted","Data":"b76b9eeee8c3e0f7c18c76ec2cbb03a52fbe065d3d50913a0edb8426f95b3ff2"}
Nov 03 16:18:44.701880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:44.701786    4964 generic.go:356] "Generic (PLEG): container finished" podID="27107370-b018-4f71-a6b0-e3266680fe97" containerID="846eeb2e6b41d7375f412fcaaa2e3662398033af926327eeac263a4d984cf2e4" exitCode=0
Nov 03 16:18:44.701880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:44.701865    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb" event={"ID":"27107370-b018-4f71-a6b0-e3266680fe97","Type":"ContainerDied","Data":"846eeb2e6b41d7375f412fcaaa2e3662398033af926327eeac263a4d984cf2e4"}
Nov 03 16:18:45.635438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.635367    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"]
Nov 03 16:18:45.639366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.639339    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.651258 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.651222    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"]
Nov 03 16:18:45.715858 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.715817    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkfd\" (UniqueName: \"kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.716332 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.716021    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.716332 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.716068    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.817276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.817231    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.817470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.817301    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.817470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.817422    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkfd\" (UniqueName: \"kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.818342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.818311    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.820669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.820641    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.831157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.831113    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkfd\" (UniqueName: \"kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd\") pod \"maas-api-revoke-sub-faecbd7a-87mx9\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:45.876112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.876079    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:45.918826 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.918776    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl\" (UniqueName: \"kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl\") pod \"27107370-b018-4f71-a6b0-e3266680fe97\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") "
Nov 03 16:18:45.918826 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.918841    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca\") pod \"27107370-b018-4f71-a6b0-e3266680fe97\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") "
Nov 03 16:18:45.919104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.918864    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca\") pod \"27107370-b018-4f71-a6b0-e3266680fe97\" (UID: \"27107370-b018-4f71-a6b0-e3266680fe97\") "
Nov 03 16:18:45.919283 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.919256    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca" pod "27107370-b018-4f71-a6b0-e3266680fe97" (UID: "27107370-b018-4f71-a6b0-e3266680fe97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:45.919701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.919671    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:45.921803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.921777    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca" pod "27107370-b018-4f71-a6b0-e3266680fe97" (UID: "27107370-b018-4f71-a6b0-e3266680fe97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:45.923756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.923727    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl" pod "27107370-b018-4f71-a6b0-e3266680fe97" (UID: "27107370-b018-4f71-a6b0-e3266680fe97"). InnerVolumeSpecName "kube-api-access-zgvbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:45.956430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:45.956378    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:46.021413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.021226    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgvbl\" (UniqueName: \"kubernetes.io/projected/27107370-b018-4f71-a6b0-e3266680fe97-kube-api-access-zgvbl\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:46.021413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.021263    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27107370-b018-4f71-a6b0-e3266680fe97-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:46.238058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.238024    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"]
Nov 03 16:18:46.240024 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:46.239993    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e0c923_8445_4bf3_ad66_81f535b1f988.slice/crio-901c52f928b3f13272079c9bccbe7e37bcb66ed2b74d909f95a7593d3322713a WatchSource:0}: Error finding container 901c52f928b3f13272079c9bccbe7e37bcb66ed2b74d909f95a7593d3322713a: Status 404 returned error can't find the container with id 901c52f928b3f13272079c9bccbe7e37bcb66ed2b74d909f95a7593d3322713a
Nov 03 16:18:46.718006 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.717969    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"
Nov 03 16:18:46.718006 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.717983    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb" event={"ID":"27107370-b018-4f71-a6b0-e3266680fe97","Type":"ContainerDied","Data":"b76b9eeee8c3e0f7c18c76ec2cbb03a52fbe065d3d50913a0edb8426f95b3ff2"}
Nov 03 16:18:46.718510 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.718034    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76b9eeee8c3e0f7c18c76ec2cbb03a52fbe065d3d50913a0edb8426f95b3ff2"
Nov 03 16:18:46.719903 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.719872    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9" event={"ID":"43e0c923-8445-4bf3-ad66-81f535b1f988","Type":"ContainerStarted","Data":"e1833f14c12893ad5690496354a800c8c74db2b78ead49a068084811f73c6b7b"}
Nov 03 16:18:46.720041 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.719930    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9" event={"ID":"43e0c923-8445-4bf3-ad66-81f535b1f988","Type":"ContainerStarted","Data":"901c52f928b3f13272079c9bccbe7e37bcb66ed2b74d909f95a7593d3322713a"}
Nov 03 16:18:46.743872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:46.743811    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9" podStartSLOduration=1.7437966249999999 podStartE2EDuration="1.743796625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:46.740751631 +0000 UTC m=+988.254133402" watchObservedRunningTime="2026-11-03 16:18:46.743796625 +0000 UTC m=+988.257178400"
Nov 03 16:18:47.729551 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:47.729511    4964 generic.go:356] "Generic (PLEG): container finished" podID="43e0c923-8445-4bf3-ad66-81f535b1f988" containerID="e1833f14c12893ad5690496354a800c8c74db2b78ead49a068084811f73c6b7b" exitCode=0
Nov 03 16:18:47.729971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:47.729597    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9" event={"ID":"43e0c923-8445-4bf3-ad66-81f535b1f988","Type":"ContainerDied","Data":"e1833f14c12893ad5690496354a800c8c74db2b78ead49a068084811f73c6b7b"}
Nov 03 16:18:48.939125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:48.939091    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:49.055853 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.055742    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca\") pod \"43e0c923-8445-4bf3-ad66-81f535b1f988\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") "
Nov 03 16:18:49.055853 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.055822    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd\" (UniqueName: \"kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd\") pod \"43e0c923-8445-4bf3-ad66-81f535b1f988\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") "
Nov 03 16:18:49.056100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.055964    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca\") pod \"43e0c923-8445-4bf3-ad66-81f535b1f988\" (UID: \"43e0c923-8445-4bf3-ad66-81f535b1f988\") "
Nov 03 16:18:49.056916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.056884    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca" pod "43e0c923-8445-4bf3-ad66-81f535b1f988" (UID: "43e0c923-8445-4bf3-ad66-81f535b1f988"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:49.059135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.059090    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca" pod "43e0c923-8445-4bf3-ad66-81f535b1f988" (UID: "43e0c923-8445-4bf3-ad66-81f535b1f988"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:49.059360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.059327    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd" pod "43e0c923-8445-4bf3-ad66-81f535b1f988" (UID: "43e0c923-8445-4bf3-ad66-81f535b1f988"). InnerVolumeSpecName "kube-api-access-cgkfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:49.158089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.158050    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:49.158089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.158088    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgkfd\" (UniqueName: \"kubernetes.io/projected/43e0c923-8445-4bf3-ad66-81f535b1f988-kube-api-access-cgkfd\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:49.158285 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.158104    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e0c923-8445-4bf3-ad66-81f535b1f988-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:49.468935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.468900    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"]
Nov 03 16:18:49.469458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.469444    4964 cpu_manager.go:414] "containerMap: removing container" podUID="43e0c923-8445-4bf3-ad66-81f535b1f988" containerName="revoke-keys"
Nov 03 16:18:49.469519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.469459    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e0c923-8445-4bf3-ad66-81f535b1f988" containerName="revoke-keys"
Nov 03 16:18:49.469519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.469489    4964 cpu_manager.go:414] "containerMap: removing container" podUID="27107370-b018-4f71-a6b0-e3266680fe97" containerName="revoke-keys"
Nov 03 16:18:49.469519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.469494    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27107370-b018-4f71-a6b0-e3266680fe97" containerName="revoke-keys"
Nov 03 16:18:49.469612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.469570    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e0c923-8445-4bf3-ad66-81f535b1f988" containerName="revoke-keys"
Nov 03 16:18:49.469612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.469582    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="27107370-b018-4f71-a6b0-e3266680fe97" containerName="revoke-keys"
Nov 03 16:18:49.473049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.473028    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.483649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.483615    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"]
Nov 03 16:18:49.562975 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.562925    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.563159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.563086    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkvtf\" (UniqueName: \"kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.563243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.563152    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.664611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.664564    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.664814 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.664672    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvtf\" (UniqueName: \"kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.664814 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.664733    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.665361 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.665341    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.667055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.667034    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.678256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.678229    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkvtf\" (UniqueName: \"kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf\") pod \"maas-api-revoke-sub-93522fa7-7zv8q\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:49.744972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.744885    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"
Nov 03 16:18:49.744972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.744908    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9" event={"ID":"43e0c923-8445-4bf3-ad66-81f535b1f988","Type":"ContainerDied","Data":"901c52f928b3f13272079c9bccbe7e37bcb66ed2b74d909f95a7593d3322713a"}
Nov 03 16:18:49.745160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.744973    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901c52f928b3f13272079c9bccbe7e37bcb66ed2b74d909f95a7593d3322713a"
Nov 03 16:18:49.789176 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:49.788665    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:50.073845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:50.073812    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"]
Nov 03 16:18:50.075449 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:50.075413    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090de332_26a4_4c2d_99cc_4e64d72f9be9.slice/crio-2ded8094ca44ab7ae6b6fd6d7d224733e57445e768deb204827173587e28fe0b WatchSource:0}: Error finding container 2ded8094ca44ab7ae6b6fd6d7d224733e57445e768deb204827173587e28fe0b: Status 404 returned error can't find the container with id 2ded8094ca44ab7ae6b6fd6d7d224733e57445e768deb204827173587e28fe0b
Nov 03 16:18:50.752956 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:50.752851    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q" event={"ID":"090de332-26a4-4c2d-99cc-4e64d72f9be9","Type":"ContainerStarted","Data":"b8e60459c785c2c3ca14b89f42340f5c363e128d2fc72059c167a3470810266e"}
Nov 03 16:18:50.752956 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:50.752917    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q" event={"ID":"090de332-26a4-4c2d-99cc-4e64d72f9be9","Type":"ContainerStarted","Data":"2ded8094ca44ab7ae6b6fd6d7d224733e57445e768deb204827173587e28fe0b"}
Nov 03 16:18:50.774920 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:50.774867    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q" podStartSLOduration=1.774852116 podStartE2EDuration="1.774852116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:50.772514173 +0000 UTC m=+992.285895950" watchObservedRunningTime="2026-11-03 16:18:50.774852116 +0000 UTC m=+992.288233891"
Nov 03 16:18:51.760541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:51.760502    4964 generic.go:356] "Generic (PLEG): container finished" podID="090de332-26a4-4c2d-99cc-4e64d72f9be9" containerID="b8e60459c785c2c3ca14b89f42340f5c363e128d2fc72059c167a3470810266e" exitCode=0
Nov 03 16:18:51.761018 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:51.760567    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q" event={"ID":"090de332-26a4-4c2d-99cc-4e64d72f9be9","Type":"ContainerDied","Data":"b8e60459c785c2c3ca14b89f42340f5c363e128d2fc72059c167a3470810266e"}
Nov 03 16:18:52.938121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:52.938097    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:53.004195 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.004148    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca\") pod \"090de332-26a4-4c2d-99cc-4e64d72f9be9\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") "
Nov 03 16:18:53.004401 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.004369    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca\") pod \"090de332-26a4-4c2d-99cc-4e64d72f9be9\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") "
Nov 03 16:18:53.004502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.004489    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf\" (UniqueName: \"kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf\") pod \"090de332-26a4-4c2d-99cc-4e64d72f9be9\" (UID: \"090de332-26a4-4c2d-99cc-4e64d72f9be9\") "
Nov 03 16:18:53.004821 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.004790    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca" pod "090de332-26a4-4c2d-99cc-4e64d72f9be9" (UID: "090de332-26a4-4c2d-99cc-4e64d72f9be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:53.006424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.006378    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca" pod "090de332-26a4-4c2d-99cc-4e64d72f9be9" (UID: "090de332-26a4-4c2d-99cc-4e64d72f9be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:53.006615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.006596    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf" pod "090de332-26a4-4c2d-99cc-4e64d72f9be9" (UID: "090de332-26a4-4c2d-99cc-4e64d72f9be9"). InnerVolumeSpecName "kube-api-access-tkvtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:53.106021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.105917    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:53.106021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.105950    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090de332-26a4-4c2d-99cc-4e64d72f9be9-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:53.106021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.105961    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkvtf\" (UniqueName: \"kubernetes.io/projected/090de332-26a4-4c2d-99cc-4e64d72f9be9-kube-api-access-tkvtf\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:53.774750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.774703    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"
Nov 03 16:18:53.774750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.774728    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q" event={"ID":"090de332-26a4-4c2d-99cc-4e64d72f9be9","Type":"ContainerDied","Data":"2ded8094ca44ab7ae6b6fd6d7d224733e57445e768deb204827173587e28fe0b"}
Nov 03 16:18:53.774962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:53.774777    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ded8094ca44ab7ae6b6fd6d7d224733e57445e768deb204827173587e28fe0b"
Nov 03 16:18:54.380331 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.380293    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"]
Nov 03 16:18:54.381002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.380982    4964 cpu_manager.go:414] "containerMap: removing container" podUID="090de332-26a4-4c2d-99cc-4e64d72f9be9" containerName="revoke-keys"
Nov 03 16:18:54.381062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.381005    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090de332-26a4-4c2d-99cc-4e64d72f9be9" containerName="revoke-keys"
Nov 03 16:18:54.381114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.381104    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="090de332-26a4-4c2d-99cc-4e64d72f9be9" containerName="revoke-keys"
Nov 03 16:18:54.388368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.388342    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.392252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.392223    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:54.393339 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.393316    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:54.406062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.405976    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"]
Nov 03 16:18:54.487255 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.487223    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"]
Nov 03 16:18:54.493893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.493865    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.501807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.501775    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"]
Nov 03 16:18:54.521788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.521750    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr58j\" (UniqueName: \"kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.521991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.521815    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.521991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.521921    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.622812 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.622770    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.622999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.622888    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8q9\" (UniqueName: \"kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.622999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.622929    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.622999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.622959    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr58j\" (UniqueName: \"kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.623107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.623008    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.623212 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.623191    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.623825 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.623801    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.625215 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.625197    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.658918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.658883    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr58j\" (UniqueName: \"kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j\") pod \"maas-api-revoke-sub-e3fe18c9-b4pst\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.702589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.702553    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:54.724140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.724069    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8q9\" (UniqueName: \"kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.724443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.724399    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.724550 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.724459    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.725246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.725222    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.727060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.727030    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.742453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.742418    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8q9\" (UniqueName: \"kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9\") pod \"maas-api-revoke-sub-3754ce8f-rptkz\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.814601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.814562    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:54.994412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:54.994364    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"]
Nov 03 16:18:54.995981 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:54.995954    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58176a27_d74c_4995_a91d_efb928cf6891.slice/crio-d56bda864ec2604032650d13da32c4bbcb5230ba02e2fedc9465f82c8fb3343a WatchSource:0}: Error finding container d56bda864ec2604032650d13da32c4bbcb5230ba02e2fedc9465f82c8fb3343a: Status 404 returned error can't find the container with id d56bda864ec2604032650d13da32c4bbcb5230ba02e2fedc9465f82c8fb3343a
Nov 03 16:18:55.109004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.108976    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"]
Nov 03 16:18:55.206007 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:55.205964    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09daff1a_856e_4756_aa29_cef67c983318.slice/crio-531119855fbb5d354964907203941705bed075890d55cf1c9eb300d434f6db58 WatchSource:0}: Error finding container 531119855fbb5d354964907203941705bed075890d55cf1c9eb300d434f6db58: Status 404 returned error can't find the container with id 531119855fbb5d354964907203941705bed075890d55cf1c9eb300d434f6db58
Nov 03 16:18:55.796489 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.796444    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst" event={"ID":"58176a27-d74c-4995-a91d-efb928cf6891","Type":"ContainerStarted","Data":"6cb6e9f9f456e9d92fb663fa0abd642c073855f16db208afaf0fda39faa191c1"}
Nov 03 16:18:55.796909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.796559    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst" event={"ID":"58176a27-d74c-4995-a91d-efb928cf6891","Type":"ContainerStarted","Data":"d56bda864ec2604032650d13da32c4bbcb5230ba02e2fedc9465f82c8fb3343a"}
Nov 03 16:18:55.798309 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.798274    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz" event={"ID":"09daff1a-856e-4756-aa29-cef67c983318","Type":"ContainerStarted","Data":"80021295ec652e6fd841275d06aa8af627bc9a4d5c74bce68c51828945c33da2"}
Nov 03 16:18:55.798492 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.798337    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz" event={"ID":"09daff1a-856e-4756-aa29-cef67c983318","Type":"ContainerStarted","Data":"531119855fbb5d354964907203941705bed075890d55cf1c9eb300d434f6db58"}
Nov 03 16:18:55.815519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.815464    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst" podStartSLOduration=1.8154440269999998 podStartE2EDuration="1.815444027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:55.81431527 +0000 UTC m=+997.327697047" watchObservedRunningTime="2026-11-03 16:18:55.815444027 +0000 UTC m=+997.328825803"
Nov 03 16:18:55.833840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:55.833781    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz" podStartSLOduration=1.833766479 podStartE2EDuration="1.833766479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:55.831757523 +0000 UTC m=+997.345139314" watchObservedRunningTime="2026-11-03 16:18:55.833766479 +0000 UTC m=+997.347148289"
Nov 03 16:18:56.805944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:56.805903    4964 generic.go:356] "Generic (PLEG): container finished" podID="09daff1a-856e-4756-aa29-cef67c983318" containerID="80021295ec652e6fd841275d06aa8af627bc9a4d5c74bce68c51828945c33da2" exitCode=0
Nov 03 16:18:56.806497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:56.805956    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz" event={"ID":"09daff1a-856e-4756-aa29-cef67c983318","Type":"ContainerDied","Data":"80021295ec652e6fd841275d06aa8af627bc9a4d5c74bce68c51828945c33da2"}
Nov 03 16:18:56.807683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:56.807657    4964 generic.go:356] "Generic (PLEG): container finished" podID="58176a27-d74c-4995-a91d-efb928cf6891" containerID="6cb6e9f9f456e9d92fb663fa0abd642c073855f16db208afaf0fda39faa191c1" exitCode=0
Nov 03 16:18:56.807811 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:56.807713    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst" event={"ID":"58176a27-d74c-4995-a91d-efb928cf6891","Type":"ContainerDied","Data":"6cb6e9f9f456e9d92fb663fa0abd642c073855f16db208afaf0fda39faa191c1"}
Nov 03 16:18:57.986144 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:57.986107    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:58.042330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.042295    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:58.065113 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.065078    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j\" (UniqueName: \"kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j\") pod \"58176a27-d74c-4995-a91d-efb928cf6891\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") "
Nov 03 16:18:58.065338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.065301    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca\") pod \"58176a27-d74c-4995-a91d-efb928cf6891\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") "
Nov 03 16:18:58.065457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.065345    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca\") pod \"58176a27-d74c-4995-a91d-efb928cf6891\" (UID: \"58176a27-d74c-4995-a91d-efb928cf6891\") "
Nov 03 16:18:58.066043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.066014    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca" pod "58176a27-d74c-4995-a91d-efb928cf6891" (UID: "58176a27-d74c-4995-a91d-efb928cf6891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:58.068310 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.068248    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca" pod "58176a27-d74c-4995-a91d-efb928cf6891" (UID: "58176a27-d74c-4995-a91d-efb928cf6891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:58.069363 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.068380    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j" pod "58176a27-d74c-4995-a91d-efb928cf6891" (UID: "58176a27-d74c-4995-a91d-efb928cf6891"). InnerVolumeSpecName "kube-api-access-tr58j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:58.167144 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167104    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9\" (UniqueName: \"kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9\") pod \"09daff1a-856e-4756-aa29-cef67c983318\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") "
Nov 03 16:18:58.167317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167254    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca\") pod \"09daff1a-856e-4756-aa29-cef67c983318\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") "
Nov 03 16:18:58.167417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167402    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca\") pod \"09daff1a-856e-4756-aa29-cef67c983318\" (UID: \"09daff1a-856e-4756-aa29-cef67c983318\") "
Nov 03 16:18:58.167745 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167718    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca" pod "09daff1a-856e-4756-aa29-cef67c983318" (UID: "09daff1a-856e-4756-aa29-cef67c983318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:58.167871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167771    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:58.167871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167793    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58176a27-d74c-4995-a91d-efb928cf6891-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:58.167871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.167808    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr58j\" (UniqueName: \"kubernetes.io/projected/58176a27-d74c-4995-a91d-efb928cf6891-kube-api-access-tr58j\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:58.169414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.169373    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca" pod "09daff1a-856e-4756-aa29-cef67c983318" (UID: "09daff1a-856e-4756-aa29-cef67c983318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:58.169460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.169440    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9" pod "09daff1a-856e-4756-aa29-cef67c983318" (UID: "09daff1a-856e-4756-aa29-cef67c983318"). InnerVolumeSpecName "kube-api-access-mk8q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:58.269329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.269292    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8q9\" (UniqueName: \"kubernetes.io/projected/09daff1a-856e-4756-aa29-cef67c983318-kube-api-access-mk8q9\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:58.269329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.269328    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09daff1a-856e-4756-aa29-cef67c983318-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:58.269329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.269338    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09daff1a-856e-4756-aa29-cef67c983318-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:58.826139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.826106    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"
Nov 03 16:18:58.826139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.826116    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst" event={"ID":"58176a27-d74c-4995-a91d-efb928cf6891","Type":"ContainerDied","Data":"d56bda864ec2604032650d13da32c4bbcb5230ba02e2fedc9465f82c8fb3343a"}
Nov 03 16:18:58.826418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.826191    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d56bda864ec2604032650d13da32c4bbcb5230ba02e2fedc9465f82c8fb3343a"
Nov 03 16:18:58.828136 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.828103    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz" event={"ID":"09daff1a-856e-4756-aa29-cef67c983318","Type":"ContainerDied","Data":"531119855fbb5d354964907203941705bed075890d55cf1c9eb300d434f6db58"}
Nov 03 16:18:58.828317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.828150    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"
Nov 03 16:18:58.828317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:58.828162    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531119855fbb5d354964907203941705bed075890d55cf1c9eb300d434f6db58"
Nov 03 16:18:59.154918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.154878    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"]
Nov 03 16:18:59.155773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.155729    4964 cpu_manager.go:414] "containerMap: removing container" podUID="58176a27-d74c-4995-a91d-efb928cf6891" containerName="revoke-keys"
Nov 03 16:18:59.155773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.155755    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58176a27-d74c-4995-a91d-efb928cf6891" containerName="revoke-keys"
Nov 03 16:18:59.155773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.155767    4964 cpu_manager.go:414] "containerMap: removing container" podUID="09daff1a-856e-4756-aa29-cef67c983318" containerName="revoke-keys"
Nov 03 16:18:59.155773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.155774    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09daff1a-856e-4756-aa29-cef67c983318" containerName="revoke-keys"
Nov 03 16:18:59.156034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.155888    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="58176a27-d74c-4995-a91d-efb928cf6891" containerName="revoke-keys"
Nov 03 16:18:59.156034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.155900    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="09daff1a-856e-4756-aa29-cef67c983318" containerName="revoke-keys"
Nov 03 16:18:59.159854 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.159832    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.163476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.163451    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:18:59.163607 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.163516    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:18:59.188157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.188121    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"]
Nov 03 16:18:59.279698 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.279582    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.279868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.279726    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.279868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.279795    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvffn\" (UniqueName: \"kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.381074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.381029    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.381291 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.381110    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.381291 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.381163    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvffn\" (UniqueName: \"kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.381828 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.381800    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.383701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.383673    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.395673 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.395644    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvffn\" (UniqueName: \"kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn\") pod \"maas-api-revoke-sub-6512d20d-l9jhg\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.473809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.473771    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:18:59.758426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.758379    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"]
Nov 03 16:18:59.759851 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:18:59.759821    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a817b7a_388a_435f_b539_31e6b936d957.slice/crio-3b76a0c516d0b8e24e3b7671f0aa3462bc59f1db84bbc20abff65f060372966b WatchSource:0}: Error finding container 3b76a0c516d0b8e24e3b7671f0aa3462bc59f1db84bbc20abff65f060372966b: Status 404 returned error can't find the container with id 3b76a0c516d0b8e24e3b7671f0aa3462bc59f1db84bbc20abff65f060372966b
Nov 03 16:18:59.836199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:18:59.836150    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg" event={"ID":"6a817b7a-388a-435f-b539-31e6b936d957","Type":"ContainerStarted","Data":"3b76a0c516d0b8e24e3b7671f0aa3462bc59f1db84bbc20abff65f060372966b"}
Nov 03 16:19:00.845088 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:00.845042    4964 generic.go:356] "Generic (PLEG): container finished" podID="6a817b7a-388a-435f-b539-31e6b936d957" containerID="397e2c63bd1ec65a1f011bb999fbbf4b357a6623af977719f02cc29a41665078" exitCode=0
Nov 03 16:19:00.845578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:00.845132    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg" event={"ID":"6a817b7a-388a-435f-b539-31e6b936d957","Type":"ContainerDied","Data":"397e2c63bd1ec65a1f011bb999fbbf4b357a6623af977719f02cc29a41665078"}
Nov 03 16:19:00.929279 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:00.929237    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:19:02.038443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.038408    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:19:02.114534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.114404    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca\") pod \"6a817b7a-388a-435f-b539-31e6b936d957\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") "
Nov 03 16:19:02.114534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.114523    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca\") pod \"6a817b7a-388a-435f-b539-31e6b936d957\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") "
Nov 03 16:19:02.114898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.114565    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn\" (UniqueName: \"kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn\") pod \"6a817b7a-388a-435f-b539-31e6b936d957\" (UID: \"6a817b7a-388a-435f-b539-31e6b936d957\") "
Nov 03 16:19:02.115154 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.115123    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca" pod "6a817b7a-388a-435f-b539-31e6b936d957" (UID: "6a817b7a-388a-435f-b539-31e6b936d957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:02.117684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.117632    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca" pod "6a817b7a-388a-435f-b539-31e6b936d957" (UID: "6a817b7a-388a-435f-b539-31e6b936d957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:02.124024 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.123979    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn" pod "6a817b7a-388a-435f-b539-31e6b936d957" (UID: "6a817b7a-388a-435f-b539-31e6b936d957"). InnerVolumeSpecName "kube-api-access-cvffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:02.215730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.215691    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:02.215730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.215727    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a817b7a-388a-435f-b539-31e6b936d957-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:02.215730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.215741    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvffn\" (UniqueName: \"kubernetes.io/projected/6a817b7a-388a-435f-b539-31e6b936d957-kube-api-access-cvffn\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:02.570930 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:19:02.570894    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:19:02.858894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.858816    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"
Nov 03 16:19:02.858894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.858847    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg" event={"ID":"6a817b7a-388a-435f-b539-31e6b936d957","Type":"ContainerDied","Data":"3b76a0c516d0b8e24e3b7671f0aa3462bc59f1db84bbc20abff65f060372966b"}
Nov 03 16:19:02.858894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:02.858894    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b76a0c516d0b8e24e3b7671f0aa3462bc59f1db84bbc20abff65f060372966b"
Nov 03 16:19:04.963411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.962714    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"]
Nov 03 16:19:04.963938 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.963575    4964 cpu_manager.go:414] "containerMap: removing container" podUID="6a817b7a-388a-435f-b539-31e6b936d957" containerName="revoke-keys"
Nov 03 16:19:04.963938 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.963604    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a817b7a-388a-435f-b539-31e6b936d957" containerName="revoke-keys"
Nov 03 16:19:04.963938 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.963746    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a817b7a-388a-435f-b539-31e6b936d957" containerName="revoke-keys"
Nov 03 16:19:04.979914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.977409    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"]
Nov 03 16:19:04.979914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.977550    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:04.985143 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.985069    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:19:04.985143 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:04.985117    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:19:05.148736 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.148693    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdrv\" (UniqueName: \"kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.148736 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.148738    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.148994 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.148807    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.249967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.249859    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdrv\" (UniqueName: \"kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.249967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.249908    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.249967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.249936    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.250837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.250801    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.252625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.252600    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.277178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.277142    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdrv\" (UniqueName: \"kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv\") pod \"maas-api-revoke-sub-c9514f47-899kg\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.314518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.314473    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:05.620740 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.620709    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"]
Nov 03 16:19:05.623045 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:05.623012    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0987bc_8d70_4214_b013_e49251b1e5a3.slice/crio-5095420abbd574f22078a8a2898e8e8766ad1f942b8b0d4a70c340e485698179 WatchSource:0}: Error finding container 5095420abbd574f22078a8a2898e8e8766ad1f942b8b0d4a70c340e485698179: Status 404 returned error can't find the container with id 5095420abbd574f22078a8a2898e8e8766ad1f942b8b0d4a70c340e485698179
Nov 03 16:19:05.881193 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:05.881106    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg" event={"ID":"eb0987bc-8d70-4214-b013-e49251b1e5a3","Type":"ContainerStarted","Data":"5095420abbd574f22078a8a2898e8e8766ad1f942b8b0d4a70c340e485698179"}
Nov 03 16:19:06.889581 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:06.889483    4964 generic.go:356] "Generic (PLEG): container finished" podID="eb0987bc-8d70-4214-b013-e49251b1e5a3" containerID="77d361434ee1a08771901288975bd03a10721489486b36170118b4c2144970ce" exitCode=0
Nov 03 16:19:06.890023 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:06.889586    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg" event={"ID":"eb0987bc-8d70-4214-b013-e49251b1e5a3","Type":"ContainerDied","Data":"77d361434ee1a08771901288975bd03a10721489486b36170118b4c2144970ce"}
Nov 03 16:19:07.284778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.284730    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"]
Nov 03 16:19:07.290028 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.290000    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.305235 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.305198    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"]
Nov 03 16:19:07.374282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.374243    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.374476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.374287    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph46x\" (UniqueName: \"kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.374568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.374510    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.475971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.475927    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.475971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.475975    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph46x\" (UniqueName: \"kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.476239 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.476044    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.476679 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.476653    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.478567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.478546    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.490145 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.490114    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph46x\" (UniqueName: \"kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x\") pod \"maas-api-revoke-sub-8a772b6e-kl4sh\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.604621 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.604533    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:07.885874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.885846    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"]
Nov 03 16:19:07.888548 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:07.888514    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3aa72a_bfb0_4a66_af13_b59eff1dd4e2.slice/crio-384ab03f5f7f47c2b4ee1e1a5b231e100016563fe8c12b3e5e2aa9b202f7888b WatchSource:0}: Error finding container 384ab03f5f7f47c2b4ee1e1a5b231e100016563fe8c12b3e5e2aa9b202f7888b: Status 404 returned error can't find the container with id 384ab03f5f7f47c2b4ee1e1a5b231e100016563fe8c12b3e5e2aa9b202f7888b
Nov 03 16:19:07.899960 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:07.899878    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh" event={"ID":"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2","Type":"ContainerStarted","Data":"384ab03f5f7f47c2b4ee1e1a5b231e100016563fe8c12b3e5e2aa9b202f7888b"}
Nov 03 16:19:08.051337 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.051308    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:08.184851 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.184818    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca\") pod \"eb0987bc-8d70-4214-b013-e49251b1e5a3\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") "
Nov 03 16:19:08.185071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.184964    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca\") pod \"eb0987bc-8d70-4214-b013-e49251b1e5a3\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") "
Nov 03 16:19:08.185071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.185007    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv\" (UniqueName: \"kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv\") pod \"eb0987bc-8d70-4214-b013-e49251b1e5a3\" (UID: \"eb0987bc-8d70-4214-b013-e49251b1e5a3\") "
Nov 03 16:19:08.185374 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.185346    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca" pod "eb0987bc-8d70-4214-b013-e49251b1e5a3" (UID: "eb0987bc-8d70-4214-b013-e49251b1e5a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:08.187247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.187223    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv" pod "eb0987bc-8d70-4214-b013-e49251b1e5a3" (UID: "eb0987bc-8d70-4214-b013-e49251b1e5a3"). InnerVolumeSpecName "kube-api-access-9jdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:08.187587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.187556    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca" pod "eb0987bc-8d70-4214-b013-e49251b1e5a3" (UID: "eb0987bc-8d70-4214-b013-e49251b1e5a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:08.287016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.286975    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:08.287016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.287008    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdrv\" (UniqueName: \"kubernetes.io/projected/eb0987bc-8d70-4214-b013-e49251b1e5a3-kube-api-access-9jdrv\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:08.287016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.287018    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0987bc-8d70-4214-b013-e49251b1e5a3-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:08.908932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.908893    4964 generic.go:356] "Generic (PLEG): container finished" podID="ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" containerID="e492b7dcea368e0b19381ed485e0fce3585589adf48d45877939559233c0ce49" exitCode=0
Nov 03 16:19:08.909363 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.908966    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh" event={"ID":"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2","Type":"ContainerDied","Data":"e492b7dcea368e0b19381ed485e0fce3585589adf48d45877939559233c0ce49"}
Nov 03 16:19:08.910460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.910437    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"
Nov 03 16:19:08.910558 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.910483    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg" event={"ID":"eb0987bc-8d70-4214-b013-e49251b1e5a3","Type":"ContainerDied","Data":"5095420abbd574f22078a8a2898e8e8766ad1f942b8b0d4a70c340e485698179"}
Nov 03 16:19:08.910558 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:08.910522    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5095420abbd574f22078a8a2898e8e8766ad1f942b8b0d4a70c340e485698179"
Nov 03 16:19:09.185544 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.185506    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"]
Nov 03 16:19:09.186303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.186285    4964 cpu_manager.go:414] "containerMap: removing container" podUID="eb0987bc-8d70-4214-b013-e49251b1e5a3" containerName="revoke-keys"
Nov 03 16:19:09.186380 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.186306    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0987bc-8d70-4214-b013-e49251b1e5a3" containerName="revoke-keys"
Nov 03 16:19:09.186458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.186446    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0987bc-8d70-4214-b013-e49251b1e5a3" containerName="revoke-keys"
Nov 03 16:19:09.190299 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.190276    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.217434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.203476    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-69cedd-openshift-default-dockercfg-5bh57\""
Nov 03 16:19:09.217434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.204028    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"]
Nov 03 16:19:09.298451 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298407    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298610 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298560    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298610 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298605    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298693    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298759    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfkx\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298805    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298849 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298825    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298945 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298930    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.298998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.298979    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.400596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.400561    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.400596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.400608    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfkx\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.400899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.400872    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.400966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.400915    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.400966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.400958    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.400975    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.401027    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.401052    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.401147    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.401195    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.401249    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.401407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.401370    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.402046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.402021    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.402151 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.402133    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.403460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.403343    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.403612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.403497    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.414830 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.414797    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfkx\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.414830 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.414822    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.506215 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.506117    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:09.799781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.799746    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"]
Nov 03 16:19:09.801432 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:09.801399    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9c1550_4b46_4a0f_b6e7_771f8691dcf2.slice/crio-9371023e31712362d0499b949b52b4c588b27d8b8fae12ee90c11bcf0f34a26f WatchSource:0}: Error finding container 9371023e31712362d0499b949b52b4c588b27d8b8fae12ee90c11bcf0f34a26f: Status 404 returned error can't find the container with id 9371023e31712362d0499b949b52b4c588b27d8b8fae12ee90c11bcf0f34a26f
Nov 03 16:19:09.804546 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.804509    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:09.804646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.804598    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:09.804725 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.804644    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:09.919906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.919866    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" event={"ID":"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2","Type":"ContainerStarted","Data":"a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947"}
Nov 03 16:19:09.920431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:09.919935    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" event={"ID":"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2","Type":"ContainerStarted","Data":"9371023e31712362d0499b949b52b4c588b27d8b8fae12ee90c11bcf0f34a26f"}
Nov 03 16:19:10.043243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.043175    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podStartSLOduration=1.043151016 podStartE2EDuration="1.043151016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:09.94828112 +0000 UTC m=+1011.461662897" watchObservedRunningTime="2026-11-03 16:19:10.043151016 +0000 UTC m=+1011.556532793"
Nov 03 16:19:10.046202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.046167    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"]
Nov 03 16:19:10.054318 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.053431    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.089643 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.089603    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"]
Nov 03 16:19:10.149213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.149166    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:10.214711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.214664    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca\") pod \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") "
Nov 03 16:19:10.214901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.214762    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca\") pod \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") "
Nov 03 16:19:10.214901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.214867    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x\" (UniqueName: \"kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x\") pod \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\" (UID: \"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2\") "
Nov 03 16:19:10.215081 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.215054    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2psz\" (UniqueName: \"kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.215170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.215124    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.215170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.215141    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca" pod "ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" (UID: "ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:10.215555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.215521    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.215693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.215621    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:10.217678 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.217654    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca" pod "ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" (UID: "ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:10.217803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.217710    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x" pod "ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" (UID: "ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2"). InnerVolumeSpecName "kube-api-access-ph46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:10.316598 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.316448    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.316598 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.316592    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2psz\" (UniqueName: \"kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.316842 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.316633    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.316842 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.316729    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:10.316842 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.316743    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph46x\" (UniqueName: \"kubernetes.io/projected/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2-kube-api-access-ph46x\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:10.317714 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.317686    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.319183 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.319158    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.340772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.340734    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2psz\" (UniqueName: \"kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz\") pod \"maas-api-revoke-sub-24ec88da-nfgkg\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.400132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.400094    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:10.513416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.507822    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:10.515480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.515065    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:10.515480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.515131    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:10.734747 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.734718    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"]
Nov 03 16:19:10.737469 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:10.737434    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20080c8e_ebd5_45b8_a274_4bfdfe08fe80.slice/crio-d7116c519ab30284e126f640c79a3303dfc8979336542bcca064b6f5bad34eec WatchSource:0}: Error finding container d7116c519ab30284e126f640c79a3303dfc8979336542bcca064b6f5bad34eec: Status 404 returned error can't find the container with id d7116c519ab30284e126f640c79a3303dfc8979336542bcca064b6f5bad34eec
Nov 03 16:19:10.948282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.948237    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg" event={"ID":"20080c8e-ebd5-45b8-a274-4bfdfe08fe80","Type":"ContainerStarted","Data":"d7116c519ab30284e126f640c79a3303dfc8979336542bcca064b6f5bad34eec"}
Nov 03 16:19:10.950534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.950498    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh" event={"ID":"ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2","Type":"ContainerDied","Data":"384ab03f5f7f47c2b4ee1e1a5b231e100016563fe8c12b3e5e2aa9b202f7888b"}
Nov 03 16:19:10.950648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.950557    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384ab03f5f7f47c2b4ee1e1a5b231e100016563fe8c12b3e5e2aa9b202f7888b"
Nov 03 16:19:10.950708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:10.950692    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"
Nov 03 16:19:11.512516 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:11.511291    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:11.512516 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:11.511355    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:11.959259 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:11.959219    4964 generic.go:356] "Generic (PLEG): container finished" podID="20080c8e-ebd5-45b8-a274-4bfdfe08fe80" containerID="87d366ddcabb050ae01c1db56bd0ddf3c2eb7fa9fb9ff414b4bea25046140ff5" exitCode=0
Nov 03 16:19:11.959763 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:11.959291    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg" event={"ID":"20080c8e-ebd5-45b8-a274-4bfdfe08fe80","Type":"ContainerDied","Data":"87d366ddcabb050ae01c1db56bd0ddf3c2eb7fa9fb9ff414b4bea25046140ff5"}
Nov 03 16:19:12.509417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:12.507885    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:12.509417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:12.507952    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:13.190180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.190152    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:13.270524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.270430    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz\" (UniqueName: \"kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz\") pod \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") "
Nov 03 16:19:13.270995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.270956    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca\") pod \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") "
Nov 03 16:19:13.271181 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.271162    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca\") pod \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\" (UID: \"20080c8e-ebd5-45b8-a274-4bfdfe08fe80\") "
Nov 03 16:19:13.272070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.272047    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca" pod "20080c8e-ebd5-45b8-a274-4bfdfe08fe80" (UID: "20080c8e-ebd5-45b8-a274-4bfdfe08fe80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:13.275096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.275072    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca" pod "20080c8e-ebd5-45b8-a274-4bfdfe08fe80" (UID: "20080c8e-ebd5-45b8-a274-4bfdfe08fe80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:13.277330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.277306    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz" pod "20080c8e-ebd5-45b8-a274-4bfdfe08fe80" (UID: "20080c8e-ebd5-45b8-a274-4bfdfe08fe80"). InnerVolumeSpecName "kube-api-access-h2psz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:13.373533 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.373477    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:13.373533 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.373528    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:13.373533 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.373537    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2psz\" (UniqueName: \"kubernetes.io/projected/20080c8e-ebd5-45b8-a274-4bfdfe08fe80-kube-api-access-h2psz\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:13.511414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.507849    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:13.511414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.507916    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:13.976668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.976629    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg" event={"ID":"20080c8e-ebd5-45b8-a274-4bfdfe08fe80","Type":"ContainerDied","Data":"d7116c519ab30284e126f640c79a3303dfc8979336542bcca064b6f5bad34eec"}
Nov 03 16:19:13.976668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.976666    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"
Nov 03 16:19:13.976893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:13.976689    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7116c519ab30284e126f640c79a3303dfc8979336542bcca064b6f5bad34eec"
Nov 03 16:19:14.507411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:14.506885    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:14.507411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:14.506966    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:15.498974 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.498932    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"]
Nov 03 16:19:15.499823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.499773    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" containerName="revoke-keys"
Nov 03 16:19:15.499823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.499798    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" containerName="revoke-keys"
Nov 03 16:19:15.499823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.499816    4964 cpu_manager.go:414] "containerMap: removing container" podUID="20080c8e-ebd5-45b8-a274-4bfdfe08fe80" containerName="revoke-keys"
Nov 03 16:19:15.499823 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.499825    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20080c8e-ebd5-45b8-a274-4bfdfe08fe80" containerName="revoke-keys"
Nov 03 16:19:15.500069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.500000    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" containerName="revoke-keys"
Nov 03 16:19:15.500069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.500013    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="20080c8e-ebd5-45b8-a274-4bfdfe08fe80" containerName="revoke-keys"
Nov 03 16:19:15.506878 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.506838    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:15.507173 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.507140    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:15.547423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.541604    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"]
Nov 03 16:19:15.547423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.541750    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.550861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.549653    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"distinct-w3-d7196a-kserve-self-signed-certs\""
Nov 03 16:19:15.600114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.600059    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.600436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.600411    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.600647 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.600601    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.600828 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.600801    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.601008 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.600993    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.601231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.601215    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxk7\" (UniqueName: \"kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.710368    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxk7\" (UniqueName: \"kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.710489    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.710532    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.710605    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.710716    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.710831    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.713412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.711485    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.717409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.716142    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.726418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.724765    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.726418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.725421    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.745662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.743214    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.749458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.745883    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxk7\" (UniqueName: \"kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7\") pod \"distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:15.869622 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:15.869586    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:16.187535 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:16.187505    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"]
Nov 03 16:19:16.191188 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:16.191160    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod300c6366_5c06_4010_a4de_313fe2680332.slice/crio-802cb54dbf28a5ca2dbf5318b22b114c2b49ab875eafb70ea7116d6e3fb2ccd4 WatchSource:0}: Error finding container 802cb54dbf28a5ca2dbf5318b22b114c2b49ab875eafb70ea7116d6e3fb2ccd4: Status 404 returned error can't find the container with id 802cb54dbf28a5ca2dbf5318b22b114c2b49ab875eafb70ea7116d6e3fb2ccd4
Nov 03 16:19:16.507157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:16.507051    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 03 16:19:16.507157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:16.507110    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 03 16:19:17.001755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.001700    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" event={"ID":"300c6366-5c06-4010-a4de-313fe2680332","Type":"ContainerStarted","Data":"b1ce72d683aa584ddd5a05f1ec40044acaf9d2db27a8e0e0d11e1a1efe46fb3f"}
Nov 03 16:19:17.002349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.002303    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" event={"ID":"300c6366-5c06-4010-a4de-313fe2680332","Type":"ContainerStarted","Data":"802cb54dbf28a5ca2dbf5318b22b114c2b49ab875eafb70ea7116d6e3fb2ccd4"}
Nov 03 16:19:17.028126 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.028071    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podStartSLOduration=2.028055925 podStartE2EDuration="2.028055925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:17.024159795 +0000 UTC m=+1018.537541563" watchObservedRunningTime="2026-11-03 16:19:17.028055925 +0000 UTC m=+1018.541437697"
Nov 03 16:19:17.512314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.512284    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:17.512762 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.512742    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:17.513505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.513483    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:19:17.870697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:17.870609    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:20.968869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.968827    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"]
Nov 03 16:19:20.974140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.974114    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:20.981201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.981169    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:19:20.981416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.981376    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:19:20.986486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.986444    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"]
Nov 03 16:19:20.994223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.993676    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:20.994223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.993825    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:20.994223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:20.993910    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk8b9\" (UniqueName: \"kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.094986 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.094943    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8b9\" (UniqueName: \"kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.095199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.095059    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.095199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.095099    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.095801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.095779    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.097530 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.097509    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.109981 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.109946    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8b9\" (UniqueName: \"kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9\") pod \"maas-api-revoke-sub-970c6349-st6gl\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.290298 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.290202    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:21.591378 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.591338    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"]
Nov 03 16:19:21.591633 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:21.591607    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10783943_763b_4d29_ba95_ad148302c624.slice/crio-b00070fe1be08014c4976bb1927cdceee6b4deb501b9b7be5ebc9c6251db4e00 WatchSource:0}: Error finding container b00070fe1be08014c4976bb1927cdceee6b4deb501b9b7be5ebc9c6251db4e00: Status 404 returned error can't find the container with id b00070fe1be08014c4976bb1927cdceee6b4deb501b9b7be5ebc9c6251db4e00
Nov 03 16:19:21.992864 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.992823    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"]
Nov 03 16:19:21.997095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:21.997058    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.006790 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.006756    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"]
Nov 03 16:19:22.043040 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.042995    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl" event={"ID":"10783943-763b-4d29-ba95-ad148302c624","Type":"ContainerStarted","Data":"eca5950ab259f88de1aeca1dda6f86df6d18b02613c736408b631c28f8cb77ea"}
Nov 03 16:19:22.043335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.043308    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl" event={"ID":"10783943-763b-4d29-ba95-ad148302c624","Type":"ContainerStarted","Data":"b00070fe1be08014c4976bb1927cdceee6b4deb501b9b7be5ebc9c6251db4e00"}
Nov 03 16:19:22.067792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.067735    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl" podStartSLOduration=2.067718109 podStartE2EDuration="2.067718109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:22.064295015 +0000 UTC m=+1023.577676792" watchObservedRunningTime="2026-11-03 16:19:22.067718109 +0000 UTC m=+1023.581099885"
Nov 03 16:19:22.110483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.110439    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9trb\" (UniqueName: \"kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.110681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.110525    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.110681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.110542    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.212253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.212162    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9trb\" (UniqueName: \"kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.212426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.212319    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.212426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.212354    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.213026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.213000    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.214913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.214889    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.225542 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.225515    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9trb\" (UniqueName: \"kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb\") pod \"maas-api-revoke-sub-036eb5f2-gtr8v\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.319569 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.319533    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:22.613649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:22.613616    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"]
Nov 03 16:19:22.617747 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:22.617717    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a257de9_1680_44a9_9acd_42eee14feb05.slice/crio-9702cedb24e5885f1d79d40bcf7f6b0e9800dde3178fd6a2d55e5d422d633b62 WatchSource:0}: Error finding container 9702cedb24e5885f1d79d40bcf7f6b0e9800dde3178fd6a2d55e5d422d633b62: Status 404 returned error can't find the container with id 9702cedb24e5885f1d79d40bcf7f6b0e9800dde3178fd6a2d55e5d422d633b62
Nov 03 16:19:23.051607 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.051556    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v" event={"ID":"5a257de9-1680-44a9-9acd-42eee14feb05","Type":"ContainerStarted","Data":"692cdc71a249c2712e5399da2829f77d835341d7f0a6456c3d19da2e3d6e4356"}
Nov 03 16:19:23.052103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.051628    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v" event={"ID":"5a257de9-1680-44a9-9acd-42eee14feb05","Type":"ContainerStarted","Data":"9702cedb24e5885f1d79d40bcf7f6b0e9800dde3178fd6a2d55e5d422d633b62"}
Nov 03 16:19:23.053302 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.053271    4964 generic.go:356] "Generic (PLEG): container finished" podID="10783943-763b-4d29-ba95-ad148302c624" containerID="eca5950ab259f88de1aeca1dda6f86df6d18b02613c736408b631c28f8cb77ea" exitCode=0
Nov 03 16:19:23.053465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.053333    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl" event={"ID":"10783943-763b-4d29-ba95-ad148302c624","Type":"ContainerDied","Data":"eca5950ab259f88de1aeca1dda6f86df6d18b02613c736408b631c28f8cb77ea"}
Nov 03 16:19:23.076622 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.076553    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v" podStartSLOduration=2.076533372 podStartE2EDuration="2.076533372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:23.076146899 +0000 UTC m=+1024.589528676" watchObservedRunningTime="2026-11-03 16:19:23.076533372 +0000 UTC m=+1024.589915151"
Nov 03 16:19:23.759614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.759508    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"]
Nov 03 16:19:23.764432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.764404    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.771836 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.771803    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b132e3-openshift-default-dockercfg-d5hc7\""
Nov 03 16:19:23.780094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.780061    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"]
Nov 03 16:19:23.832824 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.832774    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.832921    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.832985    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnm6n\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.833057    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.833090    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.833204    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833316 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.833257    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.833426    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.833527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.833462    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934194    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934230    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934270    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934323    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934355    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnm6n\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934523    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934578    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934639    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934687    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934689    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.934833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.934739    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.935235 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.935199    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.935414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.935337    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.935414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.935353    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.937103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.937079    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.937214 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.937112    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.947744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.947705    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:23.948099 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:23.948076    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnm6n\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:24.063449 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.063315    4964 generic.go:356] "Generic (PLEG): container finished" podID="5a257de9-1680-44a9-9acd-42eee14feb05" containerID="692cdc71a249c2712e5399da2829f77d835341d7f0a6456c3d19da2e3d6e4356" exitCode=0
Nov 03 16:19:24.063449 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.063429    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v" event={"ID":"5a257de9-1680-44a9-9acd-42eee14feb05","Type":"ContainerDied","Data":"692cdc71a249c2712e5399da2829f77d835341d7f0a6456c3d19da2e3d6e4356"}
Nov 03 16:19:24.081599 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.081559    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:24.273635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.273602    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:24.340944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.340848    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca\") pod \"10783943-763b-4d29-ba95-ad148302c624\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") "
Nov 03 16:19:24.340944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.340922    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca\") pod \"10783943-763b-4d29-ba95-ad148302c624\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") "
Nov 03 16:19:24.341155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.341032    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9\" (UniqueName: \"kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9\") pod \"10783943-763b-4d29-ba95-ad148302c624\" (UID: \"10783943-763b-4d29-ba95-ad148302c624\") "
Nov 03 16:19:24.341346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.341309    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca" pod "10783943-763b-4d29-ba95-ad148302c624" (UID: "10783943-763b-4d29-ba95-ad148302c624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:24.341725 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.341664    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10783943-763b-4d29-ba95-ad148302c624-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:24.344074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.344045    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca" pod "10783943-763b-4d29-ba95-ad148302c624" (UID: "10783943-763b-4d29-ba95-ad148302c624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:24.344187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.344071    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9" pod "10783943-763b-4d29-ba95-ad148302c624" (UID: "10783943-763b-4d29-ba95-ad148302c624"). InnerVolumeSpecName "kube-api-access-pk8b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:24.378471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.378381    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"]
Nov 03 16:19:24.379038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.379017    4964 cpu_manager.go:414] "containerMap: removing container" podUID="10783943-763b-4d29-ba95-ad148302c624" containerName="revoke-keys"
Nov 03 16:19:24.379127 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.379039    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10783943-763b-4d29-ba95-ad148302c624" containerName="revoke-keys"
Nov 03 16:19:24.379217 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.379204    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="10783943-763b-4d29-ba95-ad148302c624" containerName="revoke-keys"
Nov 03 16:19:24.382876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.382851    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.412141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.412098    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"]
Nov 03 16:19:24.413409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.413360    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-afb990-openshift-default-dockercfg-gc64q\""
Nov 03 16:19:24.420263 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.420229    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"]
Nov 03 16:19:24.426047 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:24.426006    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5073047c_0a5c_4f52_9be7_3c6c55c689b8.slice/crio-e7e925b5af0a1e15eff29133a598cc0b82fa9403740498a95305eb56be8a1a61 WatchSource:0}: Error finding container e7e925b5af0a1e15eff29133a598cc0b82fa9403740498a95305eb56be8a1a61: Status 404 returned error can't find the container with id e7e925b5af0a1e15eff29133a598cc0b82fa9403740498a95305eb56be8a1a61
Nov 03 16:19:24.429368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.429310    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:24.429506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.429456    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:24.429576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.429506    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:24.443350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443306    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443546 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443495    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443546 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443534    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443656 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443595    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k59w\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443656 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443624    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443653    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443761 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443720    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443792    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443827    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.443915 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443900    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10783943-763b-4d29-ba95-ad148302c624-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:24.443970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.443922    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk8b9\" (UniqueName: \"kubernetes.io/projected/10783943-763b-4d29-ba95-ad148302c624-kube-api-access-pk8b9\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:24.545259 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545215    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545518    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545595    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k59w\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545657    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545852 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545721    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545852 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545798    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.545852 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545835    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.546003 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.545882    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.546158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.546132    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.546329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.546304    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.546422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.546382    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.546608 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.546586    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.546847 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.546822    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.548597 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.548572    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.549018 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.548998    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.567871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.567834    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k59w\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.567871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.567833    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:24.699486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:24.699436    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:25.027429 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:25.027351    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b920e4_9c84_4f35_981e_1a95da93e099.slice/crio-515ad46a5f4758c7e0071df46e867294837073af5fddba3b15de94f3dcead09e WatchSource:0}: Error finding container 515ad46a5f4758c7e0071df46e867294837073af5fddba3b15de94f3dcead09e: Status 404 returned error can't find the container with id 515ad46a5f4758c7e0071df46e867294837073af5fddba3b15de94f3dcead09e
Nov 03 16:19:25.028355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.028285    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"]
Nov 03 16:19:25.037772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.037660    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:25.037772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.037758    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:25.037971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.037828    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:25.071377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.071342    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" event={"ID":"f6b920e4-9c84-4f35-981e-1a95da93e099","Type":"ContainerStarted","Data":"515ad46a5f4758c7e0071df46e867294837073af5fddba3b15de94f3dcead09e"}
Nov 03 16:19:25.073090 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.073063    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"
Nov 03 16:19:25.073230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.073088    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl" event={"ID":"10783943-763b-4d29-ba95-ad148302c624","Type":"ContainerDied","Data":"b00070fe1be08014c4976bb1927cdceee6b4deb501b9b7be5ebc9c6251db4e00"}
Nov 03 16:19:25.073230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.073133    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00070fe1be08014c4976bb1927cdceee6b4deb501b9b7be5ebc9c6251db4e00"
Nov 03 16:19:25.075904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.075872    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" event={"ID":"5073047c-0a5c-4f52-9be7-3c6c55c689b8","Type":"ContainerStarted","Data":"d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24"}
Nov 03 16:19:25.076032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.075955    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" event={"ID":"5073047c-0a5c-4f52-9be7-3c6c55c689b8","Type":"ContainerStarted","Data":"e7e925b5af0a1e15eff29133a598cc0b82fa9403740498a95305eb56be8a1a61"}
Nov 03 16:19:25.086448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.082498    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:25.088084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.086772    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:25.088084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.086832    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:25.126420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.123437    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podStartSLOduration=2.123415958 podStartE2EDuration="2.123415958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:25.116906339 +0000 UTC m=+1026.630288118" watchObservedRunningTime="2026-11-03 16:19:25.123415958 +0000 UTC m=+1026.636797734"
Nov 03 16:19:25.450635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.450594    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:25.574504 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.574463    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca\") pod \"5a257de9-1680-44a9-9acd-42eee14feb05\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") "
Nov 03 16:19:25.574969 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.574941    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca" pod "5a257de9-1680-44a9-9acd-42eee14feb05" (UID: "5a257de9-1680-44a9-9acd-42eee14feb05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:25.575071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.575055    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb\" (UniqueName: \"kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb\") pod \"5a257de9-1680-44a9-9acd-42eee14feb05\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") "
Nov 03 16:19:25.575204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.575186    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca\") pod \"5a257de9-1680-44a9-9acd-42eee14feb05\" (UID: \"5a257de9-1680-44a9-9acd-42eee14feb05\") "
Nov 03 16:19:25.576398 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.576357    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:25.578629 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.578594    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb" pod "5a257de9-1680-44a9-9acd-42eee14feb05" (UID: "5a257de9-1680-44a9-9acd-42eee14feb05"). InnerVolumeSpecName "kube-api-access-w9trb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:25.581624 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.581590    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca" pod "5a257de9-1680-44a9-9acd-42eee14feb05" (UID: "5a257de9-1680-44a9-9acd-42eee14feb05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:25.678764 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.678029    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a257de9-1680-44a9-9acd-42eee14feb05-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:25.678764 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.678075    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9trb\" (UniqueName: \"kubernetes.io/projected/5a257de9-1680-44a9-9acd-42eee14feb05-kube-api-access-w9trb\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:25.870824 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.870787    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:25.912626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:25.902697    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:26.082498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.082171    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:26.082498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.082230    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:26.088889 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.088849    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" event={"ID":"f6b920e4-9c84-4f35-981e-1a95da93e099","Type":"ContainerStarted","Data":"5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d"}
Nov 03 16:19:26.092459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.090615    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v" event={"ID":"5a257de9-1680-44a9-9acd-42eee14feb05","Type":"ContainerDied","Data":"9702cedb24e5885f1d79d40bcf7f6b0e9800dde3178fd6a2d55e5d422d633b62"}
Nov 03 16:19:26.092459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.090664    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9702cedb24e5885f1d79d40bcf7f6b0e9800dde3178fd6a2d55e5d422d633b62"
Nov 03 16:19:26.092459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.090721    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"
Nov 03 16:19:26.143124 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.143048    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podStartSLOduration=2.143026896 podStartE2EDuration="2.143026896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:26.121799818 +0000 UTC m=+1027.635181596" watchObservedRunningTime="2026-11-03 16:19:26.143026896 +0000 UTC m=+1027.656408673"
Nov 03 16:19:26.155412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.144732    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:19:26.700138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.700088    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:26.701377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.701360    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:19:26.701457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:26.701421    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 16:19:27.082368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:27.082277    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:27.082368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:27.082346    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:27.700460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:27.700416    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:19:27.700628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:27.700487    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 16:19:28.082153 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:28.082058    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:28.082153 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:28.082123    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:28.700171 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:28.700128    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:19:28.700710 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:28.700206    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 16:19:29.081963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:29.081835    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:29.081963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:29.081903    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:29.700584 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:29.700547    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:19:29.701008 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:29.700600    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 16:19:30.083048 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.082925    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:30.083230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.083040    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:30.699914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.699880    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:19:30.700091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.699933    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 16:19:30.750696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.749305    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"]
Nov 03 16:19:30.750696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.750163    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5a257de9-1680-44a9-9acd-42eee14feb05" containerName="revoke-keys"
Nov 03 16:19:30.750696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.750186    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a257de9-1680-44a9-9acd-42eee14feb05" containerName="revoke-keys"
Nov 03 16:19:30.750696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.750347    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a257de9-1680-44a9-9acd-42eee14feb05" containerName="revoke-keys"
Nov 03 16:19:30.754805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.754781    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.761469 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.761439    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"]
Nov 03 16:19:30.761634 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.761466    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"distinct-w5-69cedd-kserve-self-signed-certs\""
Nov 03 16:19:30.798439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.798372    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghv27\" (UniqueName: \"kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.798627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.798452    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.798627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.798552    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.798627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.798592    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.798627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.798616    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.798851 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.798723    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.902419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.902145    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.902419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.902263    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghv27\" (UniqueName: \"kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.902419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.902302    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.902738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.902446    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.902738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.902499    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.902738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.902531    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.906410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.903046    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.906410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.903316    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.916000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.915960    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.920156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.920081    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghv27\" (UniqueName: \"kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.953292 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.953210    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:30.976528 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:30.974334    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs\") pod \"distinct-w5-69cedd-kserve-c8b7bbccc-79kg7\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:31.075971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.074575    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:31.082434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.082377    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 16:19:31.082594 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.082459    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 16:19:31.286439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.286406    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"]
Nov 03 16:19:31.295658 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.294983    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.313466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.311664    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:19:31.313466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.312513    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:19:31.314617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.313818    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dlmz\" (UniqueName: \"kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.314617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.313990    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.314617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.314020    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.367440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.363464    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"]
Nov 03 16:19:31.427513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.425595    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.427513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.425646    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.427513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.425780    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dlmz\" (UniqueName: \"kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.427513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.427040    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.432253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.432195    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.447572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.447500    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dlmz\" (UniqueName: \"kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz\") pod \"maas-api-revoke-sub-558f91e9-rfpbl\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.460521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.460484    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"]
Nov 03 16:19:31.667990 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.667933    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:31.700402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.699861    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 16:19:31.700402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:31.699931    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 16:19:32.009437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.008834    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"]
Nov 03 16:19:32.087104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.087072    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:32.087638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.087609    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:32.088590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.088564    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:19:32.199456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.199417    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl" event={"ID":"60558c0e-271d-48fe-b286-1a39aaa92498","Type":"ContainerStarted","Data":"3edbfb391138743ba826f81cd7c81a8f923bc75e4e6396c5db0fba0d73310f3f"}
Nov 03 16:19:32.204129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.204099    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" event={"ID":"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df","Type":"ContainerStarted","Data":"65023ca49df9f0bf7f11daf40c81b27bef29809bc4b7392f50dc050d8e55e705"}
Nov 03 16:19:32.204405 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.204367    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" event={"ID":"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df","Type":"ContainerStarted","Data":"d43a9029485bbb18bfbc13408b497673c912d73eaf4b016ab15954b95eb53212"}
Nov 03 16:19:32.584177 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:19:32.584036    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:19:32.706036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.706004    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:32.707843 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.707559    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:32.708002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.707911    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:19:32.746435 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:32.746353    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podStartSLOduration=2.746333533 podStartE2EDuration="2.746333533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:32.257174712 +0000 UTC m=+1033.770556489" watchObservedRunningTime="2026-11-03 16:19:32.746333533 +0000 UTC m=+1034.259715310"
Nov 03 16:19:33.074973 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:33.074941    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:33.221589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:33.221436    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl" event={"ID":"60558c0e-271d-48fe-b286-1a39aaa92498","Type":"ContainerStarted","Data":"a3fcf9f163dc1f464b8273e8b78cc076a8b9ad894fba67acdc3ab80125c4b1da"}
Nov 03 16:19:33.255410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:33.253072    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl" podStartSLOduration=2.25305176 podStartE2EDuration="2.25305176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:33.241971268 +0000 UTC m=+1034.755353045" watchObservedRunningTime="2026-11-03 16:19:33.25305176 +0000 UTC m=+1034.766433539"
Nov 03 16:19:34.235440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:34.235398    4964 generic.go:356] "Generic (PLEG): container finished" podID="60558c0e-271d-48fe-b286-1a39aaa92498" containerID="a3fcf9f163dc1f464b8273e8b78cc076a8b9ad894fba67acdc3ab80125c4b1da" exitCode=0
Nov 03 16:19:34.235956 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:34.235503    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl" event={"ID":"60558c0e-271d-48fe-b286-1a39aaa92498","Type":"ContainerDied","Data":"a3fcf9f163dc1f464b8273e8b78cc076a8b9ad894fba67acdc3ab80125c4b1da"}
Nov 03 16:19:34.438793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:34.438749    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:19:35.449155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.449127    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:35.493959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.493920    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz\" (UniqueName: \"kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz\") pod \"60558c0e-271d-48fe-b286-1a39aaa92498\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") "
Nov 03 16:19:35.494146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.494029    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca\") pod \"60558c0e-271d-48fe-b286-1a39aaa92498\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") "
Nov 03 16:19:35.494146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.494056    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca\") pod \"60558c0e-271d-48fe-b286-1a39aaa92498\" (UID: \"60558c0e-271d-48fe-b286-1a39aaa92498\") "
Nov 03 16:19:35.494525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.494492    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca" pod "60558c0e-271d-48fe-b286-1a39aaa92498" (UID: "60558c0e-271d-48fe-b286-1a39aaa92498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:19:35.496202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.496146    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca" pod "60558c0e-271d-48fe-b286-1a39aaa92498" (UID: "60558c0e-271d-48fe-b286-1a39aaa92498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:19:35.496305 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.496272    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz" pod "60558c0e-271d-48fe-b286-1a39aaa92498" (UID: "60558c0e-271d-48fe-b286-1a39aaa92498"). InnerVolumeSpecName "kube-api-access-8dlmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:19:35.599514 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.595490    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dlmz\" (UniqueName: \"kubernetes.io/projected/60558c0e-271d-48fe-b286-1a39aaa92498-kube-api-access-8dlmz\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:35.599514 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.595534    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:35.599514 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:35.595551    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60558c0e-271d-48fe-b286-1a39aaa92498-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:19:36.278413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:36.277993    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl" event={"ID":"60558c0e-271d-48fe-b286-1a39aaa92498","Type":"ContainerDied","Data":"3edbfb391138743ba826f81cd7c81a8f923bc75e4e6396c5db0fba0d73310f3f"}
Nov 03 16:19:36.278413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:36.278075    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edbfb391138743ba826f81cd7c81a8f923bc75e4e6396c5db0fba0d73310f3f"
Nov 03 16:19:36.278413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:36.278149    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"
Nov 03 16:19:38.858802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.858764    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"]
Nov 03 16:19:38.859310 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.859297    4964 cpu_manager.go:414] "containerMap: removing container" podUID="60558c0e-271d-48fe-b286-1a39aaa92498" containerName="revoke-keys"
Nov 03 16:19:38.859348 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.859311    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60558c0e-271d-48fe-b286-1a39aaa92498" containerName="revoke-keys"
Nov 03 16:19:38.859427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.859415    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="60558c0e-271d-48fe-b286-1a39aaa92498" containerName="revoke-keys"
Nov 03 16:19:38.863114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.863096    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.870829 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.870797    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c09b17-openshift-default-dockercfg-2jjh9\""
Nov 03 16:19:38.884779 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.884747    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"]
Nov 03 16:19:38.962311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962270    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962312    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962416    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962447    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962472    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962526    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962572    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962607    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:38.962754 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:38.962709    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257q4\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.063969 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.063918    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.063988    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064035    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257q4\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064122    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064139    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064165    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064186    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064209    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064504 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064342    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064500    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064567    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064596    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064683    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.064983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.064962    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.066774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.066747    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.066944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.066926    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.077036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.077004    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.077422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.077382    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257q4\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.176868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.176832    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:39.471564 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.471521    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"]
Nov 03 16:19:39.472592 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:39.472565    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb02ee0_e67d_4d6b_bfb7_f19fd3b2b1d1.slice/crio-1ed6126faf9f6e06b71df311027ef174413e2246dd2234aa63d8d7b325dea77d WatchSource:0}: Error finding container 1ed6126faf9f6e06b71df311027ef174413e2246dd2234aa63d8d7b325dea77d: Status 404 returned error can't find the container with id 1ed6126faf9f6e06b71df311027ef174413e2246dd2234aa63d8d7b325dea77d
Nov 03 16:19:39.474971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.474911    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:39.475082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.475027    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:39.475082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:39.475073    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:19:40.313879 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:40.313827    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" event={"ID":"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1","Type":"ContainerStarted","Data":"3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586"}
Nov 03 16:19:40.314474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:40.313907    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" event={"ID":"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1","Type":"ContainerStarted","Data":"1ed6126faf9f6e06b71df311027ef174413e2246dd2234aa63d8d7b325dea77d"}
Nov 03 16:19:40.360632 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:40.360563    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podStartSLOduration=2.36054376 podStartE2EDuration="2.36054376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:40.359722167 +0000 UTC m=+1041.873103944" watchObservedRunningTime="2026-11-03 16:19:40.36054376 +0000 UTC m=+1041.873925538"
Nov 03 16:19:41.075373 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:41.075335    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:41.095841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:41.095810    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:41.177190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:41.177149    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:41.179166 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:41.179128    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 16:19:41.179293 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:41.179195    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 16:19:41.361521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:41.361433    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:19:42.178178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:42.178130    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 16:19:42.178421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:42.178204    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 16:19:43.177786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:43.177750    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 16:19:43.178218 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:43.177808    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 16:19:44.191951 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:44.191903    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 16:19:44.192525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:44.191969    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 16:19:45.177874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:45.177834    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 16:19:45.178095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:45.177894    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 16:19:46.178378 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:46.178335    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 16:19:46.178802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:46.178426    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 16:19:47.182324 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:47.182292    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:47.182812 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:47.182652    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:47.183356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:47.183330    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:19:49.569750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.569712    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"]
Nov 03 16:19:49.575202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.575176    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.586512 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.586480    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"]
Nov 03 16:19:49.587627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.587602    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7196a\"/\"distinct-2-w3-d7196a-kserve-self-signed-certs\""
Nov 03 16:19:49.748150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.748105    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.748371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.748162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.748371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.748204    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.748371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.748255    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.748371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.748312    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.748617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.748381    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5l4j\" (UniqueName: \"kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.849699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.849598    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5l4j\" (UniqueName: \"kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.849894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.849730    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.849894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.849753    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.849894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.849776    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.849894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.849818    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.849894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.849853    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.850244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.850226    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.852872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.852799    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.852872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.852816    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.853870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.853229    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.856231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.856201    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.863625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.863594    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5l4j\" (UniqueName: \"kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j\") pod \"distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") " pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:49.894611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:49.894578    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:50.209741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:50.209702    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"]
Nov 03 16:19:50.210231 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:50.210185    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59bf2433_3ac6_4208_b49c_0cfea8472d5e.slice/crio-a6c9851963dfa408968821f0214bc577a9d753f2221f10500ec6541c715cc011 WatchSource:0}: Error finding container a6c9851963dfa408968821f0214bc577a9d753f2221f10500ec6541c715cc011: Status 404 returned error can't find the container with id a6c9851963dfa408968821f0214bc577a9d753f2221f10500ec6541c715cc011
Nov 03 16:19:50.424151 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:50.424103    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" event={"ID":"59bf2433-3ac6-4208-b49c-0cfea8472d5e","Type":"ContainerStarted","Data":"adc8768542b57d3824e49cccd1de30438e153f8389031ed2f06e91873fc68a5d"}
Nov 03 16:19:50.424336 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:50.424159    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" event={"ID":"59bf2433-3ac6-4208-b49c-0cfea8472d5e","Type":"ContainerStarted","Data":"a6c9851963dfa408968821f0214bc577a9d753f2221f10500ec6541c715cc011"}
Nov 03 16:19:50.451705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:50.451644    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podStartSLOduration=1.451625822 podStartE2EDuration="1.451625822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:50.446930516 +0000 UTC m=+1051.960312293" watchObservedRunningTime="2026-11-03 16:19:50.451625822 +0000 UTC m=+1051.965007620"
Nov 03 16:19:50.895373 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:50.895326    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:57.356960 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.356915    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"]
Nov 03 16:19:57.363249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.363217    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.367227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.367195    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:19:57.367896 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.367874    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:19:57.368717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.368692    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"]
Nov 03 16:19:57.417293 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.417249    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zc78\" (UniqueName: \"kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.417498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.417309    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.417567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.417542    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.518555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.518524    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.518753 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.518633    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.518807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.518783    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc78\" (UniqueName: \"kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.519475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.519444    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.521344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.521315    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.533503 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.533466    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc78\" (UniqueName: \"kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78\") pod \"maas-api-revoke-sub-9b34fdc2-l94vq\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.683709 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.683659    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:19:57.975630 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:19:57.975595    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b588ca2_7a14_4690_a6eb_2b0ffcc8640c.slice/crio-d2f9e6aa4fd8deeee3d475a27a9345e3911f30e62500f92953f8db75ff7ab226 WatchSource:0}: Error finding container d2f9e6aa4fd8deeee3d475a27a9345e3911f30e62500f92953f8db75ff7ab226: Status 404 returned error can't find the container with id d2f9e6aa4fd8deeee3d475a27a9345e3911f30e62500f92953f8db75ff7ab226
Nov 03 16:19:57.976241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:57.976208    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"]
Nov 03 16:19:58.538599 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:58.538560    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq" event={"ID":"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c","Type":"ContainerStarted","Data":"71e91d1ef8e459bf9ae847f21f030d391395b66b1417c47bfbabdb8e1559aac9"}
Nov 03 16:19:58.539093 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:58.538624    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq" event={"ID":"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c","Type":"ContainerStarted","Data":"d2f9e6aa4fd8deeee3d475a27a9345e3911f30e62500f92953f8db75ff7ab226"}
Nov 03 16:19:58.559741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:58.559692    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq" podStartSLOduration=1.55967872 podStartE2EDuration="1.55967872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:58.557669451 +0000 UTC m=+1060.071051228" watchObservedRunningTime="2026-11-03 16:19:58.55967872 +0000 UTC m=+1060.073060495"
Nov 03 16:19:59.550184 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:59.550143    4964 generic.go:356] "Generic (PLEG): container finished" podID="2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" containerID="71e91d1ef8e459bf9ae847f21f030d391395b66b1417c47bfbabdb8e1559aac9" exitCode=0
Nov 03 16:19:59.550751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:59.550200    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq" event={"ID":"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c","Type":"ContainerDied","Data":"71e91d1ef8e459bf9ae847f21f030d391395b66b1417c47bfbabdb8e1559aac9"}
Nov 03 16:19:59.897297 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:59.895696    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:19:59.912793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:19:59.912761    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:20:00.570704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.570671    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:20:00.765210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.765173    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:20:00.865630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.865537    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78\" (UniqueName: \"kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78\") pod \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") "
Nov 03 16:20:00.865815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.865688    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca\") pod \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") "
Nov 03 16:20:00.865874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.865853    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca\") pod \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\" (UID: \"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c\") "
Nov 03 16:20:00.866306 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.866235    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca" pod "2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" (UID: "2b588ca2-7a14-4690-a6eb-2b0ffcc8640c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:00.866532 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.866510    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:00.867885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.867853    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca" pod "2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" (UID: "2b588ca2-7a14-4690-a6eb-2b0ffcc8640c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:00.868022 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.868006    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78" pod "2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" (UID: "2b588ca2-7a14-4690-a6eb-2b0ffcc8640c"). InnerVolumeSpecName "kube-api-access-9zc78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:00.967817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.967774    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zc78\" (UniqueName: \"kubernetes.io/projected/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-kube-api-access-9zc78\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:00.967817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:00.967809    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:01.567221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:01.567190    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"
Nov 03 16:20:01.567461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:01.567194    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq" event={"ID":"2b588ca2-7a14-4690-a6eb-2b0ffcc8640c","Type":"ContainerDied","Data":"d2f9e6aa4fd8deeee3d475a27a9345e3911f30e62500f92953f8db75ff7ab226"}
Nov 03 16:20:01.567461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:01.567330    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f9e6aa4fd8deeee3d475a27a9345e3911f30e62500f92953f8db75ff7ab226"
Nov 03 16:20:02.567425 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:02.566833    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:20:08.490474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.490438    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"]
Nov 03 16:20:08.491048 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.491033    4964 cpu_manager.go:414] "containerMap: removing container" podUID="2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" containerName="revoke-keys"
Nov 03 16:20:08.491091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.491048    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" containerName="revoke-keys"
Nov 03 16:20:08.491160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.491151    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" containerName="revoke-keys"
Nov 03 16:20:08.495211 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.495184    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.512034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.512001    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"kube-root-ca.crt\""
Nov 03 16:20:08.512347 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.512329    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"openshift-service-ca.crt\""
Nov 03 16:20:08.514319 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.513095    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"distinct-2-w6-afb990-kserve-self-signed-certs\""
Nov 03 16:20:08.514319 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.513199    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"default-dockercfg-z6l22\""
Nov 03 16:20:08.519557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.519522    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"]
Nov 03 16:20:08.565421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.565343    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.565421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.565414    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626sr\" (UniqueName: \"kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.565643 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.565572    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.565688 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.565636    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.565832 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.565809    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.565881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.565856    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.666695 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.666663    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.666906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.666732    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.666906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.666754    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626sr\" (UniqueName: \"kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.666906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.666804    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.666906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.666840    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.667145 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.666964    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.667205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.667179    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.667265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.667238    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.667525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.667502    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.669453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.669380    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.669655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.669635    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.682839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.681089    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626sr\" (UniqueName: \"kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr\") pod \"distinct-2-w6-afb990-kserve-5b6b488987-h2kfm\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.819052 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.818927    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:08.838723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.838670    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"]
Nov 03 16:20:08.845011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.844141    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.857684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.857352    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"]
Nov 03 16:20:08.970255 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970166    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970255 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970212    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970304    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970426    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970505    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqcq4\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970702 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970582    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970702 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970614    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970702 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970660    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:08.970849 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:08.970699    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.044773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.044726    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"]
Nov 03 16:20:09.049257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.049226    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.055678 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.055633    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4a19cd-openshift-default-dockercfg-h4cvm\""
Nov 03 16:20:09.061438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.061405    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"]
Nov 03 16:20:09.072091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072017    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072054    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072098    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072124    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072187    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072229    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072268    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072554 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072327    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072554 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072423    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqcq4\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072752    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.072850 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072811    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.073016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.072971    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.073218 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.073189    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.073315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.073247    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.080745 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.080713    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.080848    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.085797 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.085768    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.085923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.085837    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqcq4\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.141189 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.141155    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"]
Nov 03 16:20:09.143489 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:09.143455    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a6b2b1_84b1_43df_8495_85c3d904a0d7.slice/crio-b29b1d390eb08ff77c7deef6773ce6e60a8a337c3de06c1e69ff045b9f5257da WatchSource:0}: Error finding container b29b1d390eb08ff77c7deef6773ce6e60a8a337c3de06c1e69ff045b9f5257da: Status 404 returned error can't find the container with id b29b1d390eb08ff77c7deef6773ce6e60a8a337c3de06c1e69ff045b9f5257da
Nov 03 16:20:09.174238 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174106    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174238 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174155    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174317    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174448    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174478    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174527    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174591    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqc85\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174670    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.174802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.174743    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.189466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.189434    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:09.278956 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.278826    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.277223    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.279664    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.279739    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.279848    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.279922    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.279949    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.280007    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.280033    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc85\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.280064    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.280612    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.280663    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.281413    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.281939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.281165    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.287040 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.286983    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.291081 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.291049    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.300755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.300054    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc85\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.300755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.300068    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.309237 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.309201    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"]
Nov 03 16:20:09.328504 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.328425    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.342332 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.340266    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"]
Nov 03 16:20:09.354770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.354569    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"]
Nov 03 16:20:09.354770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.354689    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.356416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.356372    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"]
Nov 03 16:20:09.366703 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.366678    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.483865    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.483933    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.483978    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95wd\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484014    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484085    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484135    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484165    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484245    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484303    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484338    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwb8c\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484434    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484459    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.484571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484524    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.485350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484681    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.485350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484718    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.485350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.484967    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.485350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.485006    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.485350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.485032    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.545381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.545349    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"]
Nov 03 16:20:09.547039 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:09.547007    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae60f3d_5fff_427c_8917_ecc52cc8929b.slice/crio-91737a807b312ed0933475b3497fc2ae48849f35bb1cfb011f046a8a0edec46e WatchSource:0}: Error finding container 91737a807b312ed0933475b3497fc2ae48849f35bb1cfb011f046a8a0edec46e: Status 404 returned error can't find the container with id 91737a807b312ed0933475b3497fc2ae48849f35bb1cfb011f046a8a0edec46e
Nov 03 16:20:09.550246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.550210    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:09.550343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.550287    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:09.550343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.550330    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:09.586716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.586716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586667    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.586716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586743    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586771    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586832    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586881    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.586963    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587002 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587001    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587043    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95wd\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587065    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587084    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587158    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587185    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587193    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587260    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587322    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587366    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587375    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587424    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwb8c\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587542    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587659    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.587933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587803    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.587933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.587931    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.588147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.588005    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.588214 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.588174    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.588316 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.588284    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.590074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.589681    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.590226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.590205    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.590342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.590322    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.590716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.590696    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.619955 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.619923    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.628637 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.628606    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.628776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.628711    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95wd\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.628776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.628720    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwb8c\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.647665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.647626    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" event={"ID":"2ae60f3d-5fff-427c-8917-ecc52cc8929b","Type":"ContainerStarted","Data":"5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb"}
Nov 03 16:20:09.647953 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.647690    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" event={"ID":"2ae60f3d-5fff-427c-8917-ecc52cc8929b","Type":"ContainerStarted","Data":"91737a807b312ed0933475b3497fc2ae48849f35bb1cfb011f046a8a0edec46e"}
Nov 03 16:20:09.649977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.649943    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" event={"ID":"e8a6b2b1-84b1-43df-8495-85c3d904a0d7","Type":"ContainerStarted","Data":"d514fc8d7a0d69b6be01137bfeb591437c97582ac7bcb40f8492f2ad3fc0243d"}
Nov 03 16:20:09.650105 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.649988    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" event={"ID":"e8a6b2b1-84b1-43df-8495-85c3d904a0d7","Type":"ContainerStarted","Data":"b29b1d390eb08ff77c7deef6773ce6e60a8a337c3de06c1e69ff045b9f5257da"}
Nov 03 16:20:09.657780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.657747    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:09.672938 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.672895    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"]
Nov 03 16:20:09.673315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.673103    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:09.702562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.691748    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.702562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.692770    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"]
Nov 03 16:20:09.702562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.700624    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:09.702562 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.700813    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:09.812039 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.811969    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podStartSLOduration=1.811943795 podStartE2EDuration="1.811943795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:09.694767148 +0000 UTC m=+1071.208148924" watchObservedRunningTime="2026-11-03 16:20:09.811943795 +0000 UTC m=+1071.325325573"
Nov 03 16:20:09.826152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.826071    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.826298 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.826162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sr2\" (UniqueName: \"kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.833031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.829212    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:09.833031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.829405    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.833031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.831336    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:09.833031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.831442    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:09.833031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.831491    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:09.872488 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.871112    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"]
Nov 03 16:20:09.893867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.890564    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podStartSLOduration=1.890540992 podStartE2EDuration="1.890540992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:09.717516216 +0000 UTC m=+1071.230897993" watchObservedRunningTime="2026-11-03 16:20:09.890540992 +0000 UTC m=+1071.403922769"
Nov 03 16:20:09.929535 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.929500    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"]
Nov 03 16:20:09.933354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.931182    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.933354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.931250    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sr2\" (UniqueName: \"kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.933354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.931408    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.933354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.932211    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.940758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.940725    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"]
Nov 03 16:20:09.941236 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.941150    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:09.948935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.948899    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:09.954724 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:09.954694    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sr2\" (UniqueName: \"kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2\") pod \"maas-api-revoke-sub-55985def-x7hsk\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:10.033178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033140    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033182    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033210    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033356    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033398    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033494    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033534    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033666    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgxv\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.033742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.033711    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.111271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.111232    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"]
Nov 03 16:20:10.114467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.113048    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"]
Nov 03 16:20:10.115300 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:10.115271    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a800b84_3440_4c9a_9084_5ff793a7082a.slice/crio-a4e8768ab8b7bac721e6fb13ef1d32b181b908e554d28feb27e3dd92b1eda309 WatchSource:0}: Error finding container a4e8768ab8b7bac721e6fb13ef1d32b181b908e554d28feb27e3dd92b1eda309: Status 404 returned error can't find the container with id a4e8768ab8b7bac721e6fb13ef1d32b181b908e554d28feb27e3dd92b1eda309
Nov 03 16:20:10.124206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.117655    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.124206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.117742    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.124206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.117803    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.130523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.130476    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.130798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.130774    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.131258 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.131235    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.135602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135544    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.135733 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135607    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.135733 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135646    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.135850 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135765    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.135850 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135806    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.135949 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135868    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.135949 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135895    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.136055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.135996    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgxv\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.136055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.136042    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.136602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.136570    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.136845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.136825    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.137044 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.137028    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.137612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.137586    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.141721 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.138298    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.141721 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.141134    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.154668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.154634    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.157199 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.157166    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgxv\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.158093 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.158065    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.190630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.190606    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:10.192578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.192545    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:10.192758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.192600    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:10.195103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.195012    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:10.364946 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.364631    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:10.777480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.777443    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"]
Nov 03 16:20:10.778301 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.777970    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" event={"ID":"15070bd9-7432-4a73-80c0-f9cd4af75282","Type":"ContainerStarted","Data":"f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9"}
Nov 03 16:20:10.778301 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.778024    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" event={"ID":"15070bd9-7432-4a73-80c0-f9cd4af75282","Type":"ContainerStarted","Data":"97a2264f1f4fb743c69b2e76de4027401156c80fe6a82945b33b30663f922376"}
Nov 03 16:20:10.780630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.780606    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" event={"ID":"6a800b84-3440-4c9a-9084-5ff793a7082a","Type":"ContainerStarted","Data":"fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a"}
Nov 03 16:20:10.781806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.781781    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" event={"ID":"6a800b84-3440-4c9a-9084-5ff793a7082a","Type":"ContainerStarted","Data":"a4e8768ab8b7bac721e6fb13ef1d32b181b908e554d28feb27e3dd92b1eda309"}
Nov 03 16:20:10.784887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.784867    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" event={"ID":"242af026-10c3-4aba-baec-ef113d93d527","Type":"ContainerStarted","Data":"13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de"}
Nov 03 16:20:10.785027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.785016    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" event={"ID":"242af026-10c3-4aba-baec-ef113d93d527","Type":"ContainerStarted","Data":"5f518b8e97777fb636991c734bc6265bdc6efea57e0d448a6303509b815d7dee"}
Nov 03 16:20:10.866934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.866865    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podStartSLOduration=1.8668413149999998 podStartE2EDuration="1.866841315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:10.80236981 +0000 UTC m=+1072.315751587" watchObservedRunningTime="2026-11-03 16:20:10.866841315 +0000 UTC m=+1072.380223094"
Nov 03 16:20:10.887464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.887408    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podStartSLOduration=1.887376217 podStartE2EDuration="1.887376217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:10.845732193 +0000 UTC m=+1072.359113970" watchObservedRunningTime="2026-11-03 16:20:10.887376217 +0000 UTC m=+1072.400757993"
Nov 03 16:20:10.895539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.895469    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podStartSLOduration=1.895445414 podStartE2EDuration="1.895445414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:10.886948596 +0000 UTC m=+1072.400330375" watchObservedRunningTime="2026-11-03 16:20:10.895445414 +0000 UTC m=+1072.408827190"
Nov 03 16:20:10.966180 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:10.966146    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58b8929_76d9_4bf5_89d8_a8904d1c3366.slice/crio-841959e190f2a2daeb5299ea9af1cb0c7d38cac90cfa29cd47cf1caef6312f0f WatchSource:0}: Error finding container 841959e190f2a2daeb5299ea9af1cb0c7d38cac90cfa29cd47cf1caef6312f0f: Status 404 returned error can't find the container with id 841959e190f2a2daeb5299ea9af1cb0c7d38cac90cfa29cd47cf1caef6312f0f
Nov 03 16:20:10.975718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.974900    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.975718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.974993    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.975718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.975041    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:10.980551 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:10.977910    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"]
Nov 03 16:20:11.192669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.192636    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:11.192838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.192692    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:11.374856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.374759    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:11.403486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.399531    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:11.403486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.399614    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:11.667415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.667303    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:11.667785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.667754    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:11.668011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.667983    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:11.679654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.679412    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:11.679654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.679543    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:11.679654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.679591    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:11.903726 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.903684    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk" event={"ID":"db8192ee-f489-4aff-9b92-a4b476f61ca1","Type":"ContainerStarted","Data":"e3a0e1d348ea391ff55f08944fbd2e13ba1621aff0bcbc945bb218bbf3a9e735"}
Nov 03 16:20:11.904358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.904329    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk" event={"ID":"db8192ee-f489-4aff-9b92-a4b476f61ca1","Type":"ContainerStarted","Data":"39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34"}
Nov 03 16:20:11.920759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.920667    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" event={"ID":"e58b8929-76d9-4bf5-89d8-a8904d1c3366","Type":"ContainerStarted","Data":"34812bf92e76f534571ada6c536e7f506276400aacacff72cea2519a09ae389a"}
Nov 03 16:20:11.921056 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.921032    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" event={"ID":"e58b8929-76d9-4bf5-89d8-a8904d1c3366","Type":"ContainerStarted","Data":"841959e190f2a2daeb5299ea9af1cb0c7d38cac90cfa29cd47cf1caef6312f0f"}
Nov 03 16:20:11.943410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:11.943326    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk" podStartSLOduration=2.943301421 podStartE2EDuration="2.943301421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:11.929835022 +0000 UTC m=+1073.443216811" watchObservedRunningTime="2026-11-03 16:20:11.943301421 +0000 UTC m=+1073.456683199"
Nov 03 16:20:12.017932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.017868    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podStartSLOduration=3.017845003 podStartE2EDuration="3.017845003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:11.976294367 +0000 UTC m=+1073.489676158" watchObservedRunningTime="2026-11-03 16:20:12.017845003 +0000 UTC m=+1073.531226780"
Nov 03 16:20:12.208437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.207585    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:12.208437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.207648    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:12.356616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.356559    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:12.375662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.375626    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:12.376454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.375893    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:12.378125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.378102    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:12.380241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.380123    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:12.380241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.380177    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:12.475918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.451992    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:12.662673 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.662631    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:12.682289 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.682245    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:12.682983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.678937    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:12.683179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.683156    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:12.984982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:12.984944    4964 generic.go:356] "Generic (PLEG): container finished" podID="db8192ee-f489-4aff-9b92-a4b476f61ca1" containerID="e3a0e1d348ea391ff55f08944fbd2e13ba1621aff0bcbc945bb218bbf3a9e735" exitCode=0
Nov 03 16:20:13.000101 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.000062    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk" event={"ID":"db8192ee-f489-4aff-9b92-a4b476f61ca1","Type":"ContainerDied","Data":"e3a0e1d348ea391ff55f08944fbd2e13ba1621aff0bcbc945bb218bbf3a9e735"}
Nov 03 16:20:13.028726 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.028684    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:20:13.076419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.075934    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:13.118750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.118698    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:13.195838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.195800    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:13.196117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.196082    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:13.237195 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.237097    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:20:13.251101 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.251054    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:20:13.343917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.339307    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.364812    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.365264    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.365654    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.367931    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.367969    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.368233    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:13.368410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.368263    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:13.541744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.541542    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:13.578245 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.566917    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:20:13.639427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.638516    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:13.775411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.771611    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:13.775411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.771679    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:13.775411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.771773    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:13.775411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.771799    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:13.992553 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.988924    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:20:13.992553 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:13.990333    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Nov 03 16:20:14.180411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.086329    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:14.278026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.262048    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:20:14.278026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.117502    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:14.278026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.182040    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:20:14.325501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.278448    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:20:14.325501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.290244    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:20:14.325501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.290316    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:20:14.325501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.219614    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:14.325501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.290862    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:14.325501 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.182079    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:20:14.365431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.345135    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:14.378491 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.373559    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:14.378785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.374234    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:14.378939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.374280    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:20:14.455661 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.381794    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:14.455906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.382138    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:20:14.456095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.456077    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:20:14.472319 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.383031    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:14.472643 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.472598    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:14.655021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.654972    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:14.750640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.750601    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:14.750954 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.750917    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:14.754976 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.754945    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:14.755257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.755217    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:14.827642 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.827605    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:20:14.980091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:14.941979    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:20:15.013322 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.013260    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 03 16:20:15.103598 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.103539    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.262084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.261963    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:15.269506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.262327    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:15.455411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.370354    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.500016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.376741    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:15.502180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.376787    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:15.503323 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.376832    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.520233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.485780    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.585414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.572184    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:20:15.588366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.499298    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.589151 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.499414    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:15.589362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.589335    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:15.589548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.572041    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:15.589725 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.589697    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:15.612667 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.499131    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.688826 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.688786    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:15.699519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.699467    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:20:15.701130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.701094    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:20:15.757425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.742437    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:15.757756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.757714    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:15.817466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.802468    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:15.817466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:15.802544    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:16.015283 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.015238    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:16.096983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.096890    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:20:16.133423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.120506    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:16.134206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.134165    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:20:16.211797 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.209689    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:16.211797 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.209762    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:16.407412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.381557    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:16.407412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.381624    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:16.407412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.382221    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:16.407412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.382263    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:16.548412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.512948    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:16.548412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.532428    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:20:16.548412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.547200    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:16.548412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.547595    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:20:16.548412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.547908    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:20:16.625466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.590157    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:16.826413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.679835    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:16.919463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.915928    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.707639    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.709543    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.879515    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.707602    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.709890    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.946318    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:16.985603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:16.985091    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:20:17.123807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.103289    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.135429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.135367    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.197556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.190113    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:17.197556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.190180    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:17.367169 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.367129    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:17.367490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.367457    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:17.374417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.367965    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:17.374752 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.374710    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:17.523481 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.516102    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.542413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.536001    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.633532 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.633474    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.634221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.634189    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:17.650424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.646557    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.650424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.649553    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:17.669436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.669327    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:17.669653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.669449    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:17.681422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.680754    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:17.681422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.680823    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:17.730410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.700884    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:20:17.771403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.771340    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:20:17.846528 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.846432    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca\") pod \"db8192ee-f489-4aff-9b92-a4b476f61ca1\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") "
Nov 03 16:20:17.846864 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.846840    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2\" (UniqueName: \"kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2\") pod \"db8192ee-f489-4aff-9b92-a4b476f61ca1\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") "
Nov 03 16:20:17.876747 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.874631    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca\") pod \"db8192ee-f489-4aff-9b92-a4b476f61ca1\" (UID: \"db8192ee-f489-4aff-9b92-a4b476f61ca1\") "
Nov 03 16:20:17.894097 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.894041    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 03 16:20:17.906706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.906629    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca" pod "db8192ee-f489-4aff-9b92-a4b476f61ca1" (UID: "db8192ee-f489-4aff-9b92-a4b476f61ca1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:17.947587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.947521    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.991413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.949021    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:20:17.992308 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.991517    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:17.992867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:17.992835    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:20:18.095414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.090945    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:18.181424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.176504    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:18.181424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.177934    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:20:18.229436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.194533    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:20:18.229436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.195161    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:18.229436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.195231    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:18.278456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.267522    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca" pod "db8192ee-f489-4aff-9b92-a4b476f61ca1" (UID: "db8192ee-f489-4aff-9b92-a4b476f61ca1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:18.394878 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.394830    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2" pod "db8192ee-f489-4aff-9b92-a4b476f61ca1" (UID: "db8192ee-f489-4aff-9b92-a4b476f61ca1"). InnerVolumeSpecName "kube-api-access-57sr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:18.395529 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.395503    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:18.395766 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.395740    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:18.426421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.413273    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57sr2\" (UniqueName: \"kubernetes.io/projected/db8192ee-f489-4aff-9b92-a4b476f61ca1-kube-api-access-57sr2\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:18.427990 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.427934    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db8192ee-f489-4aff-9b92-a4b476f61ca1-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:18.428178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.427634    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:18.434467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.428347    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:18.558728 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.558680    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:18.663418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.659338    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:18.683424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.681560    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:18.712523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.689919    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:18.712523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.689976    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:18.712523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.697893    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:20:18.712523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.698315    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:18.767465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.766452    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:20:18.898418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:18.893358    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.053737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.053643    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.054338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.054307    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:20:19.055614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.055587    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.082442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.082400    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:19.177455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.177346    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.200140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.178074    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.200140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.191350    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:20:19.229410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.202548    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:19.229410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.202602    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:19.397462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.383555    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:19.397735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.397695    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:19.410417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.386568    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:19.410711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.410676    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:19.464420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.454230    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:19.465252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.465232    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:20:19.467223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.465111    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.561987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.561946    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:20:19.576065 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.567289    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"
Nov 03 16:20:19.595609 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.595558    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk" event={"ID":"db8192ee-f489-4aff-9b92-a4b476f61ca1","Type":"ContainerDied","Data":"39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34"}
Nov 03 16:20:19.595899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.595872    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34"
Nov 03 16:20:19.668705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.668668    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:19.674746 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.673850    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:19.683411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.679337    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:19.758632 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.684031    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:19.817410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.749648    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.818170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.776548    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:19.849197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.849157    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:20:19.892670 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:19.892629    4964 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:20.099767 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.099670    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:20.100350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.100313    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:20:20.104694 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.104668    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:20:20.123531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.111014    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:20.186064 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.186016    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:20.193959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.193919    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:20.194204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.194175    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:20.194359 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.194341    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:20:20.399648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.397785    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:20.399648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.397847    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:20.399648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.398838    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:20.399648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.398879    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:20.484224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.482056    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:20.484224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.482623    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:20:20.579593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.573478    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:20.687550 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.687518    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:20.687874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.687827    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:20.703453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.687511    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:20.703453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.697051    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:20.828766 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.828704    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:20.868749 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.868679    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:20.871357 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.871327    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:20:20.922487 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.922418    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Nov 03 16:20:20.961794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:20.961697    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:20:21.004434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.000947    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:21.152670 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.152631    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:20:21.176373 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.176320    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:20:21.197937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.197882    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:21.274726 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.211796    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:21.387727 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.387680    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:21.388035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.387995    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:21.450561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.447659    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:21.450561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.447736    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:21.512024 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.511982    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:21.512294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.512270    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:21.618723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.618611    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:21.792413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.709294    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:20:21.814735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.814693    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:20:21.816620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.789630    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:21.816783 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.816681    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:21.816847 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.816804    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:20:21.870480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.794066    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:21.870480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.854093    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:21.907493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.891497    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:21.935412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:21.899378    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.036247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.036205    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.083414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.065570    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:20:22.084410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.084361    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:20:22.200084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.167554    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:20:22.200084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.168120    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:20:22.200084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.198806    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.200630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.200601    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:22.200819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.200794    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:22.214932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.214828    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:20:22.247010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.246572    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.335215 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.335171    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.385412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.384401    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:20:22.385412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.384482    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:20:22.496472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.468138    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:22.496472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.468207    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:22.596523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.596471    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:22.632407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.598821    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:22.706477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.682763    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:22.706477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.706336    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:22.706477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.683066    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:22.706477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.706401    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:22.706477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.698734    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:20:22.816469 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.816276    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.818332 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.817454    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.864619 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.862466    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:20:22.944411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.938487    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:22.944411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:22.939464    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:20:23.046215 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.046159    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.085975 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.085603    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:20:23.195651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.195613    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:20:23.195937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.195904    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:20:23.225829 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.208513    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.253807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.252780    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.287422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.273995    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.344515 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.339647    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:20:23.449411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.418903    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:23.460480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.451529    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:23.598442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.598318    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.612412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.601529    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:20:23.668410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.660685    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:20:23.668410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.660749    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:20:23.682804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.682768    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 16:20:23.707409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.701960    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 16:20:23.707409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.703111    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:20:23.712417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.712348    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.716663 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.715556    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:20:23.822408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.817683    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.864465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.863511    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:23.864465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.864315    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:20:23.966411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:23.965577    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.055508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.055458    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:20:24.193029 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.192999    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:24.254408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.254143    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.294471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.292486    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.294471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.293240    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:20:24.345009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.344965    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.363267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.363237    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:20:24.396651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.394494    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:24.396651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.394555    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:24.406841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.404080    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:24.406841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.404668    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:20:24.406841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.404738    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.406841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.405448    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:20:24.691411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.689405    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:24.691411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.689490    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:24.716342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.715625    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.736750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.724172    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:24.789458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.782275    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:20:24.804242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.793885    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:20:24.819128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.818130    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:24.933409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:24.931490    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:20:25.070261 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.070164    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:25.262473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.259899    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:20:25.276423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.274476    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:20:25.276423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.274560    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:25.305674 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.304746    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:20:25.430827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.430690    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:25.442802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.441679    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:25.443305 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.443278    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 16:20:25.443549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.443521    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 16:20:25.474992 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.472492    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:20:25.474992 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.472563    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:25.499774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.499276    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:20:25.499774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.499718    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:20:25.654636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.654595    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:20:25.985751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.985706    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:25.989321 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:25.989295    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:20:26.111429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:26.110677    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:26.143775 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:26.143732    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:20:26.293538 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:26.293430    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:20:26.529175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:26.529144    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:26.530475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:26.530451    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:20:28.833487 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:28.833455    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:28.847824 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:28.847785    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:20:30.477462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.477417    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"]
Nov 03 16:20:30.478093 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.478075    4964 cpu_manager.go:414] "containerMap: removing container" podUID="db8192ee-f489-4aff-9b92-a4b476f61ca1" containerName="revoke-keys"
Nov 03 16:20:30.478142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.478094    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8192ee-f489-4aff-9b92-a4b476f61ca1" containerName="revoke-keys"
Nov 03 16:20:30.478223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.478214    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8192ee-f489-4aff-9b92-a4b476f61ca1" containerName="revoke-keys"
Nov 03 16:20:30.483105 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.483084    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.487230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.487205    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:30.488062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.488040    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:30.503869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.503837    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"]
Nov 03 16:20:30.603112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.603078    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.603303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.603135    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpprn\" (UniqueName: \"kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.603401 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.603358    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.704921 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.704881    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.705112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.705068    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.705192 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.705176    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpprn\" (UniqueName: \"kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.705867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.705842    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.707350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.707326    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.717523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.717495    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpprn\" (UniqueName: \"kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn\") pod \"maas-api-revoke-sub-dc2c76c0-rrjwg\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:30.798030 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:30.797939    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:31.121514 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:31.121484    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"]
Nov 03 16:20:31.126611 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:31.126575    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0230724_c182_423c_bca6_542e675034bc.slice/crio-ff505976c39837a75600dabdccb12b008617c12e3e2064158d933065e6723bf4 WatchSource:0}: Error finding container ff505976c39837a75600dabdccb12b008617c12e3e2064158d933065e6723bf4: Status 404 returned error can't find the container with id ff505976c39837a75600dabdccb12b008617c12e3e2064158d933065e6723bf4
Nov 03 16:20:31.550189 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:31.550140    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg" event={"ID":"a0230724-c182-423c-bca6-542e675034bc","Type":"ContainerStarted","Data":"9fd83ef8c0cc93a55e568255d4ac7a3c0552b994e46aabd88a309f284aa7646e"}
Nov 03 16:20:31.550630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:31.550215    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg" event={"ID":"a0230724-c182-423c-bca6-542e675034bc","Type":"ContainerStarted","Data":"ff505976c39837a75600dabdccb12b008617c12e3e2064158d933065e6723bf4"}
Nov 03 16:20:31.597580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:31.597517    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg" podStartSLOduration=1.59749906 podStartE2EDuration="1.59749906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:31.588680933 +0000 UTC m=+1093.102062710" watchObservedRunningTime="2026-11-03 16:20:31.59749906 +0000 UTC m=+1093.110880859"
Nov 03 16:20:32.562320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:32.562286    4964 generic.go:356] "Generic (PLEG): container finished" podID="a0230724-c182-423c-bca6-542e675034bc" containerID="9fd83ef8c0cc93a55e568255d4ac7a3c0552b994e46aabd88a309f284aa7646e" exitCode=0
Nov 03 16:20:32.562785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:32.562373    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg" event={"ID":"a0230724-c182-423c-bca6-542e675034bc","Type":"ContainerDied","Data":"9fd83ef8c0cc93a55e568255d4ac7a3c0552b994e46aabd88a309f284aa7646e"}
Nov 03 16:20:32.573237 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:32.573190    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:20:33.767928 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.767898    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:33.864112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.864028    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca\") pod \"a0230724-c182-423c-bca6-542e675034bc\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") "
Nov 03 16:20:33.864270 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.864185    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca\") pod \"a0230724-c182-423c-bca6-542e675034bc\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") "
Nov 03 16:20:33.864311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.864274    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn\" (UniqueName: \"kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn\") pod \"a0230724-c182-423c-bca6-542e675034bc\" (UID: \"a0230724-c182-423c-bca6-542e675034bc\") "
Nov 03 16:20:33.864525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.864500    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca" pod "a0230724-c182-423c-bca6-542e675034bc" (UID: "a0230724-c182-423c-bca6-542e675034bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:33.864817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.864796    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0230724-c182-423c-bca6-542e675034bc-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:33.866459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.866379    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca" pod "a0230724-c182-423c-bca6-542e675034bc" (UID: "a0230724-c182-423c-bca6-542e675034bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:33.866459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.866443    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn" pod "a0230724-c182-423c-bca6-542e675034bc" (UID: "a0230724-c182-423c-bca6-542e675034bc"). InnerVolumeSpecName "kube-api-access-rpprn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:33.966406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.966358    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpprn\" (UniqueName: \"kubernetes.io/projected/a0230724-c182-423c-bca6-542e675034bc-kube-api-access-rpprn\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:33.966406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:33.966411    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0230724-c182-423c-bca6-542e675034bc-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:34.582592 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:34.582560    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"
Nov 03 16:20:34.582592 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:34.582577    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg" event={"ID":"a0230724-c182-423c-bca6-542e675034bc","Type":"ContainerDied","Data":"ff505976c39837a75600dabdccb12b008617c12e3e2064158d933065e6723bf4"}
Nov 03 16:20:34.582822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:34.582625    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff505976c39837a75600dabdccb12b008617c12e3e2064158d933065e6723bf4"
Nov 03 16:20:34.847521 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:34.847358    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:20:36.464639 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:36.464584    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:20:37.217548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.217487    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:37.316905 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.316842    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:37.523609 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.523490    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:20:37.756541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.756487    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:20:37.757248 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.756797    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:37.822020 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.821895    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:37.846737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.846690    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:37.868614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:37.868557    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:38.063538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:38.063485    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:38.089167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:38.089064    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:38.525027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:38.524980    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:38.526424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:38.526399    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:20:38.584147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:38.584116    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:20:39.152033 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.151991    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"]
Nov 03 16:20:39.152988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.152958    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a0230724-c182-423c-bca6-542e675034bc" containerName="revoke-keys"
Nov 03 16:20:39.152988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.152988    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0230724-c182-423c-bca6-542e675034bc" containerName="revoke-keys"
Nov 03 16:20:39.153233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.153215    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0230724-c182-423c-bca6-542e675034bc" containerName="revoke-keys"
Nov 03 16:20:39.157713 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.157686    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.177758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.177725    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"]
Nov 03 16:20:39.186901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.186863    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.186909    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187026    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187057    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187118    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187159    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187327 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187311    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187373 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187343    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.187479 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.187464    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.238813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.238770    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"]
Nov 03 16:20:39.243749 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.243723    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.259788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.259728    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7196a-openshift-default-dockercfg-wxfhk\""
Nov 03 16:20:39.265595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.265549    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"]
Nov 03 16:20:39.290062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290015    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290084    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290127    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97k2c\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290174    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290235    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290258    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290300    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290337    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290426    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290469    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290501    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290525    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290573    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290597    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290642    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290734    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.290967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290760    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.290967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.290831    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.291710 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.291684    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.292793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.292765    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.295160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.293364    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.295160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.293605    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.295160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.294032    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.295160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.294544    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.295615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.295593    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.306567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.306528    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.307175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.307143    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.392004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.391972    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392023    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392054    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392076    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392295    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392336    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392435    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392451    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392549 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392538    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392686 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392580    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392742 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392685    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97k2c\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392686    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392904    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.392998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.392927    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.394829 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.394803    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.395041 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.395025    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.407205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.407114    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.407205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.407192    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97k2c\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.477123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.477086    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:39.574547 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.572731    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:39.786895 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:39.786858    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d37ae_a202_4576_9c03_f041c43410df.slice/crio-05e4b16e7be7bb453ca77cc4c4cd237763e6077e1389b571df3dde04ad27204b WatchSource:0}: Error finding container 05e4b16e7be7bb453ca77cc4c4cd237763e6077e1389b571df3dde04ad27204b: Status 404 returned error can't find the container with id 05e4b16e7be7bb453ca77cc4c4cd237763e6077e1389b571df3dde04ad27204b
Nov 03 16:20:39.788928 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.788815    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"]
Nov 03 16:20:39.790104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.790063    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:39.790180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.790151    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:39.790222 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.790192    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:39.880513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.880478    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"]
Nov 03 16:20:39.880672 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:39.880603    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027fe49a_1c99_41a6_bbd3_c5a324f9b4ec.slice/crio-8da291c72949e7e0d13e52036f155a10cfb6a4edac4f29fdaa1f2d4c9da7235d WatchSource:0}: Error finding container 8da291c72949e7e0d13e52036f155a10cfb6a4edac4f29fdaa1f2d4c9da7235d: Status 404 returned error can't find the container with id 8da291c72949e7e0d13e52036f155a10cfb6a4edac4f29fdaa1f2d4c9da7235d
Nov 03 16:20:39.883360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.883325    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:39.883467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.883412    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:39.883467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:39.883441    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:40.745335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:40.745223    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" event={"ID":"246d37ae-a202-4576-9c03-f041c43410df","Type":"ContainerStarted","Data":"1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd"}
Nov 03 16:20:40.745335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:40.745300    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" event={"ID":"246d37ae-a202-4576-9c03-f041c43410df","Type":"ContainerStarted","Data":"05e4b16e7be7bb453ca77cc4c4cd237763e6077e1389b571df3dde04ad27204b"}
Nov 03 16:20:40.747425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:40.747372    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" event={"ID":"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec","Type":"ContainerStarted","Data":"18fd0293c44e884aa41416961a4f18750aabf623ef33fdf3e6258a49bd4b8700"}
Nov 03 16:20:40.747557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:40.747454    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" event={"ID":"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec","Type":"ContainerStarted","Data":"8da291c72949e7e0d13e52036f155a10cfb6a4edac4f29fdaa1f2d4c9da7235d"}
Nov 03 16:20:40.773540 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:40.773482    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podStartSLOduration=1.7734679610000001 podStartE2EDuration="1.773467961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:40.772609002 +0000 UTC m=+1102.285990778" watchObservedRunningTime="2026-11-03 16:20:40.773467961 +0000 UTC m=+1102.286849737"
Nov 03 16:20:40.826407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:40.803611    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podStartSLOduration=1.803589812 podStartE2EDuration="1.803589812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:40.79894459 +0000 UTC m=+1102.312326369" watchObservedRunningTime="2026-11-03 16:20:40.803589812 +0000 UTC m=+1102.316971589"
Nov 03 16:20:41.273657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.273622    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"]
Nov 03 16:20:41.279357 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.279329    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.284053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.284028    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:41.284198 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.284103    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:41.369539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.365527    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"]
Nov 03 16:20:41.369539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.368468    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zmp\" (UniqueName: \"kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.369539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.368647    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.369539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.368729    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.472461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.472410    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.472775 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.472748    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.473625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.473599    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zmp\" (UniqueName: \"kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.474790 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.474757    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.476419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.476375    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.477313 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.477293    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:41.478813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.478785    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:20:41.478914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.478839    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:20:41.487284 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.487250    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zmp\" (UniqueName: \"kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp\") pod \"maas-api-revoke-sub-97434216-75dxk\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:41.573916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.573843    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:41.575649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.575621    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:20:41.575793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.575698    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:20:41.594517 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:41.594485    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:42.074828 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.074785    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"]
Nov 03 16:20:42.080957 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:42.080677    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5065088f_eec5_4e2a_81af_689a0a58972e.slice/crio-b2bf914a04c7a4c3e232584cc9458b2cb4f745b79251ff9e9bc1ef1e6437d890 WatchSource:0}: Error finding container b2bf914a04c7a4c3e232584cc9458b2cb4f745b79251ff9e9bc1ef1e6437d890: Status 404 returned error can't find the container with id b2bf914a04c7a4c3e232584cc9458b2cb4f745b79251ff9e9bc1ef1e6437d890
Nov 03 16:20:42.478722 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.478681    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:20:42.478901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.478765    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:20:42.574371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.574333    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:20:42.574630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.574601    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:20:42.784381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.784096    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk" event={"ID":"5065088f-eec5-4e2a-81af-689a0a58972e","Type":"ContainerStarted","Data":"92dfdc5ae8b97ba217cc611e98f648e1bcaa310829140eca331bfaffba30a881"}
Nov 03 16:20:42.784381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:42.784147    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk" event={"ID":"5065088f-eec5-4e2a-81af-689a0a58972e","Type":"ContainerStarted","Data":"b2bf914a04c7a4c3e232584cc9458b2cb4f745b79251ff9e9bc1ef1e6437d890"}
Nov 03 16:20:43.478335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:43.478283    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:20:43.478760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:43.478341    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:20:43.574255 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:43.574218    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:20:43.574435 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:43.574284    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:20:43.793738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:43.793641    4964 generic.go:356] "Generic (PLEG): container finished" podID="5065088f-eec5-4e2a-81af-689a0a58972e" containerID="92dfdc5ae8b97ba217cc611e98f648e1bcaa310829140eca331bfaffba30a881" exitCode=0
Nov 03 16:20:43.793738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:43.793732    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk" event={"ID":"5065088f-eec5-4e2a-81af-689a0a58972e","Type":"ContainerDied","Data":"92dfdc5ae8b97ba217cc611e98f648e1bcaa310829140eca331bfaffba30a881"}
Nov 03 16:20:44.480567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:44.478281    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:20:44.480567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:44.478344    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:20:44.582413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:44.579041    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:20:44.582413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:44.579134    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:20:45.206730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.206645    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:45.296994 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.296955    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca\") pod \"5065088f-eec5-4e2a-81af-689a0a58972e\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") "
Nov 03 16:20:45.297161 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.297122    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp\" (UniqueName: \"kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp\") pod \"5065088f-eec5-4e2a-81af-689a0a58972e\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") "
Nov 03 16:20:45.297213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.297184    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca\") pod \"5065088f-eec5-4e2a-81af-689a0a58972e\" (UID: \"5065088f-eec5-4e2a-81af-689a0a58972e\") "
Nov 03 16:20:45.298206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.298156    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca" pod "5065088f-eec5-4e2a-81af-689a0a58972e" (UID: "5065088f-eec5-4e2a-81af-689a0a58972e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:45.300348 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.300101    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca" pod "5065088f-eec5-4e2a-81af-689a0a58972e" (UID: "5065088f-eec5-4e2a-81af-689a0a58972e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:45.300500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.300475    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp" pod "5065088f-eec5-4e2a-81af-689a0a58972e" (UID: "5065088f-eec5-4e2a-81af-689a0a58972e"). InnerVolumeSpecName "kube-api-access-z4zmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:45.399640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.399606    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zmp\" (UniqueName: \"kubernetes.io/projected/5065088f-eec5-4e2a-81af-689a0a58972e-kube-api-access-z4zmp\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:45.399640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.399636    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:45.399640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.399646    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5065088f-eec5-4e2a-81af-689a0a58972e-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:45.477737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.477650    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:20:45.477737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.477716    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:20:45.574044 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.574008    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:20:45.574548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.574470    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:20:45.879938 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.879836    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk" event={"ID":"5065088f-eec5-4e2a-81af-689a0a58972e","Type":"ContainerDied","Data":"b2bf914a04c7a4c3e232584cc9458b2cb4f745b79251ff9e9bc1ef1e6437d890"}
Nov 03 16:20:45.880242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.880221    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2bf914a04c7a4c3e232584cc9458b2cb4f745b79251ff9e9bc1ef1e6437d890"
Nov 03 16:20:45.880340 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:45.880109    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"
Nov 03 16:20:46.478147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:46.478099    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:20:46.478372 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:46.478156    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:20:46.573465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:46.573421    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:20:46.573649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:46.573506    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:20:46.639411 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:46.638420    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache]"
Nov 03 16:20:48.078411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.077588    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:48.078950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.078699    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:48.079568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.079539    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:20:48.133298 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.133258    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"]
Nov 03 16:20:48.134082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.134011    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5065088f-eec5-4e2a-81af-689a0a58972e" containerName="revoke-keys"
Nov 03 16:20:48.134082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.134080    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5065088f-eec5-4e2a-81af-689a0a58972e" containerName="revoke-keys"
Nov 03 16:20:48.134333 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.134313    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5065088f-eec5-4e2a-81af-689a0a58972e" containerName="revoke-keys"
Nov 03 16:20:48.139458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.139205    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.149372 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.149343    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:48.153125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.152503    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:48.169058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.168639    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"]
Nov 03 16:20:48.193870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.192523    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:48.199451 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.198555    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:48.199913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.199891    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:20:48.218349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.215727    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.218349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.215826    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.218349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.216022    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzc8j\" (UniqueName: \"kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.318329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.318282    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.321906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.320373    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.327456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.318344    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.327749 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.327729    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzc8j\" (UniqueName: \"kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.341517 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.341444    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzc8j\" (UniqueName: \"kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.341722 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.341614    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-397f76bc-rz7zk\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.473690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.473654    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:48.775541 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:48.775510    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78310d4_be52_437f_8059_e2ea0b2c6564.slice/crio-08b81abd93447256e3c439e018c8fae1d484b055695c9074ef31b614c0477761 WatchSource:0}: Error finding container 08b81abd93447256e3c439e018c8fae1d484b055695c9074ef31b614c0477761: Status 404 returned error can't find the container with id 08b81abd93447256e3c439e018c8fae1d484b055695c9074ef31b614c0477761
Nov 03 16:20:48.775689 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.775663    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"]
Nov 03 16:20:48.910445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:48.910379    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk" event={"ID":"b78310d4-be52-437f-8059-e2ea0b2c6564","Type":"ContainerStarted","Data":"08b81abd93447256e3c439e018c8fae1d484b055695c9074ef31b614c0477761"}
Nov 03 16:20:49.772552 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:49.772513    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:20:49.928865 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:49.928366    4964 generic.go:356] "Generic (PLEG): container finished" podID="b78310d4-be52-437f-8059-e2ea0b2c6564" containerID="2452daea98e87432c8ffc390035c8737d0a16f77b74695598f2edfebc60ca856" exitCode=0
Nov 03 16:20:49.928865 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:49.928441    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk" event={"ID":"b78310d4-be52-437f-8059-e2ea0b2c6564","Type":"ContainerDied","Data":"2452daea98e87432c8ffc390035c8737d0a16f77b74695598f2edfebc60ca856"}
Nov 03 16:20:51.115245 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.115214    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:51.172993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.172945    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca\") pod \"b78310d4-be52-437f-8059-e2ea0b2c6564\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") "
Nov 03 16:20:51.175213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.175174    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca" pod "b78310d4-be52-437f-8059-e2ea0b2c6564" (UID: "b78310d4-be52-437f-8059-e2ea0b2c6564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:51.274676 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.274592    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j\" (UniqueName: \"kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j\") pod \"b78310d4-be52-437f-8059-e2ea0b2c6564\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") "
Nov 03 16:20:51.274676 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.274654    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca\") pod \"b78310d4-be52-437f-8059-e2ea0b2c6564\" (UID: \"b78310d4-be52-437f-8059-e2ea0b2c6564\") "
Nov 03 16:20:51.275110 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.275082    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca" pod "b78310d4-be52-437f-8059-e2ea0b2c6564" (UID: "b78310d4-be52-437f-8059-e2ea0b2c6564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:51.275178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.275096    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:51.276855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.276822    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j" pod "b78310d4-be52-437f-8059-e2ea0b2c6564" (UID: "b78310d4-be52-437f-8059-e2ea0b2c6564"). InnerVolumeSpecName "kube-api-access-xzc8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:51.377011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.376967    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzc8j\" (UniqueName: \"kubernetes.io/projected/b78310d4-be52-437f-8059-e2ea0b2c6564-kube-api-access-xzc8j\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:51.377011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.377009    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78310d4-be52-437f-8059-e2ea0b2c6564-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:51.950147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.950115    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"
Nov 03 16:20:51.950147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.950129    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk" event={"ID":"b78310d4-be52-437f-8059-e2ea0b2c6564","Type":"ContainerDied","Data":"08b81abd93447256e3c439e018c8fae1d484b055695c9074ef31b614c0477761"}
Nov 03 16:20:51.950362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:51.950208    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b81abd93447256e3c439e018c8fae1d484b055695c9074ef31b614c0477761"
Nov 03 16:20:53.705263 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.705219    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"]
Nov 03 16:20:53.706104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.706078    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b78310d4-be52-437f-8059-e2ea0b2c6564" containerName="revoke-keys"
Nov 03 16:20:53.706184 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.706105    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78310d4-be52-437f-8059-e2ea0b2c6564" containerName="revoke-keys"
Nov 03 16:20:53.706313 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.706300    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78310d4-be52-437f-8059-e2ea0b2c6564" containerName="revoke-keys"
Nov 03 16:20:53.710962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.710934    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.717919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.716139    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:20:53.717919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.716727    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:20:53.721066 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.721025    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"]
Nov 03 16:20:53.811976 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.811938    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.812193 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.811990    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zncl\" (UniqueName: \"kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.812193 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.812163    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.913978 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.913935    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zncl\" (UniqueName: \"kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.914171 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.914031    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.914325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.914266    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.914935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.914900    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.916892 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.916864    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:53.926559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:53.926531    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zncl\" (UniqueName: \"kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl\") pod \"maas-api-revoke-sub-ec2909aa-c7wq2\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:54.023933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.023844    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"]
Nov 03 16:20:54.029142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.029113    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.041563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.041533    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:54.054733 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.054702    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"]
Nov 03 16:20:54.147485 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147443    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147666 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147500    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147666 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147542    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147666 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147578    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147666 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147623    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147672    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147706    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147787    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.147914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.147835    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jmz\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249017 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.248979    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249043    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249571    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249630    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249715 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jmz\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249789 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249774    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249846 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249806    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249846    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249881    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.249991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.249933    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.250697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.250673    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.250968 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.250945    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.252161 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.252133    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.252826 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.252802    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.254162 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.254119    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.254356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.254332    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.270047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.270021    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.270190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.270167    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jmz\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.409239 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.409198    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:54.420505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.420474    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"]
Nov 03 16:20:54.574869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.574831    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"]
Nov 03 16:20:54.582061 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.582031    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.652182 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.648606    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"]
Nov 03 16:20:54.658700 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.655413    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"]
Nov 03 16:20:54.662108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.661051    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667025    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667074    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667198    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667279    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667312    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667522    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667729    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667758    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.667785    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpbw\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.679437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.672479    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"]
Nov 03 16:20:54.712744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.712709    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"]
Nov 03 16:20:54.716760 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:54.716702    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0538eea0_4223_42cf_88ac_648f96b8fe8e.slice/crio-be440bcf787697fad3b9f102887622bd1f4b035a22fb6dc010370a1b73944b91 WatchSource:0}: Error finding container be440bcf787697fad3b9f102887622bd1f4b035a22fb6dc010370a1b73944b91: Status 404 returned error can't find the container with id be440bcf787697fad3b9f102887622bd1f4b035a22fb6dc010370a1b73944b91
Nov 03 16:20:54.721000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.720934    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:54.721129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.721101    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:54.721196 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.721166    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:54.769052 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769014    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.769209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769097    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.769209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769139    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngmd\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.769302 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769272    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.769346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769329    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.769425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769357    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.769425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769395    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpbw\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.769531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769433    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.769531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769458    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.769531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769487    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.769660 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769531    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769754    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769842    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769856    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.769940    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770000    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770021    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770127    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770204    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770244    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.770448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770405    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.771031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770602    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.771031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.770851    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.773165 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.773145    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.773370 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.773347    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.783135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.783103    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpbw\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.783437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.783379    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.872344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872307    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.872595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872365    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.872595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872561    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.872819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872797    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.872966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872950    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.873081 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.873065    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.873183 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.873170    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngmd\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.873362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.873341    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.873598 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.873579    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.873964    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872833    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.872819    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.874514    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.875276    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.875644    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.877047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.875738    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.885773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.885743    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngmd\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.885773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.885755    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:54.901642 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.901594    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:54.981339 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.979750    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" event={"ID":"0538eea0-4223-42cf-88ac-648f96b8fe8e","Type":"ContainerStarted","Data":"a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842"}
Nov 03 16:20:54.981339 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.979821    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" event={"ID":"0538eea0-4223-42cf-88ac-648f96b8fe8e","Type":"ContainerStarted","Data":"be440bcf787697fad3b9f102887622bd1f4b035a22fb6dc010370a1b73944b91"}
Nov 03 16:20:54.986424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.984420    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2" event={"ID":"76cb7287-9039-47a4-9afa-e3f310eb7c26","Type":"ContainerStarted","Data":"5ab99ed62a9e81a5c3c031398bb3d7fdb75e7604c08f298ce43fc56825adb244"}
Nov 03 16:20:54.986424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.984486    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2" event={"ID":"76cb7287-9039-47a4-9afa-e3f310eb7c26","Type":"ContainerStarted","Data":"2fc1a08a573e3e9f1e11a359f729ee1d89aa8f329bb14e2bfc107067b13a761d"}
Nov 03 16:20:54.992667 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:54.992585    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:55.031702 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.031637    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podStartSLOduration=2.031578982 podStartE2EDuration="2.031578982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:55.030253652 +0000 UTC m=+1116.543635430" watchObservedRunningTime="2026-11-03 16:20:55.031578982 +0000 UTC m=+1116.544960752"
Nov 03 16:20:55.068473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.060861    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2" podStartSLOduration=2.060835233 podStartE2EDuration="2.060835233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:55.059693132 +0000 UTC m=+1116.573074923" watchObservedRunningTime="2026-11-03 16:20:55.060835233 +0000 UTC m=+1116.574217008"
Nov 03 16:20:55.225441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.225311    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"]
Nov 03 16:20:55.234441 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:55.234375    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774f3575_ea70_41a8_92bd_df711b8eb36c.slice/crio-27714edadd17d5719130084c5e596581d32cc8d55ff37f502afd3c66cba3bffa WatchSource:0}: Error finding container 27714edadd17d5719130084c5e596581d32cc8d55ff37f502afd3c66cba3bffa: Status 404 returned error can't find the container with id 27714edadd17d5719130084c5e596581d32cc8d55ff37f502afd3c66cba3bffa
Nov 03 16:20:55.238463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.238366    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:55.238574 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.238521    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:55.238621 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.238569    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:55.345536 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.345193    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"]
Nov 03 16:20:55.348525 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:20:55.348493    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe45f54_0726_4d25_a8c5_1f582fa55534.slice/crio-34490c632312622f245bf341248f4d498468ff5b7f3578172fb271cf6f0225af WatchSource:0}: Error finding container 34490c632312622f245bf341248f4d498468ff5b7f3578172fb271cf6f0225af: Status 404 returned error can't find the container with id 34490c632312622f245bf341248f4d498468ff5b7f3578172fb271cf6f0225af
Nov 03 16:20:55.351967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.351918    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:55.352076 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.352018    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:55.352076 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.352062    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:20:55.413898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.413781    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:20:55.429967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.427981    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:20:55.429967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:55.428045    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:20:56.000449 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.000414    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" event={"ID":"1fe45f54-0726-4d25-a8c5-1f582fa55534","Type":"ContainerStarted","Data":"030c7e0791831af202fa956dbeb4fa6fad61e8cfdb4ff9fb760d400fb8b27728"}
Nov 03 16:20:56.000988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.000962    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" event={"ID":"1fe45f54-0726-4d25-a8c5-1f582fa55534","Type":"ContainerStarted","Data":"34490c632312622f245bf341248f4d498468ff5b7f3578172fb271cf6f0225af"}
Nov 03 16:20:56.010408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.002766    4964 generic.go:356] "Generic (PLEG): container finished" podID="76cb7287-9039-47a4-9afa-e3f310eb7c26" containerID="5ab99ed62a9e81a5c3c031398bb3d7fdb75e7604c08f298ce43fc56825adb244" exitCode=0
Nov 03 16:20:56.010408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.002884    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2" event={"ID":"76cb7287-9039-47a4-9afa-e3f310eb7c26","Type":"ContainerDied","Data":"5ab99ed62a9e81a5c3c031398bb3d7fdb75e7604c08f298ce43fc56825adb244"}
Nov 03 16:20:56.021274 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.021156    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" event={"ID":"774f3575-ea70-41a8-92bd-df711b8eb36c","Type":"ContainerStarted","Data":"446c4e69c52c2a76d7ba95e6211136008e2aca9396c9075151a748cd24d2f93a"}
Nov 03 16:20:56.021274 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.021218    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" event={"ID":"774f3575-ea70-41a8-92bd-df711b8eb36c","Type":"ContainerStarted","Data":"27714edadd17d5719130084c5e596581d32cc8d55ff37f502afd3c66cba3bffa"}
Nov 03 16:20:56.085579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.075569    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podStartSLOduration=2.075543546 podStartE2EDuration="2.075543546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.074816351 +0000 UTC m=+1117.588198159" watchObservedRunningTime="2026-11-03 16:20:56.075543546 +0000 UTC m=+1117.588925323"
Nov 03 16:20:56.187816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.185471    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podStartSLOduration=2.1854521 podStartE2EDuration="2.1854521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.180850555 +0000 UTC m=+1117.694232351" watchObservedRunningTime="2026-11-03 16:20:56.1854521 +0000 UTC m=+1117.698833877"
Nov 03 16:20:56.422415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.412823    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:20:56.422415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.412907    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:20:56.907241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.905995    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:20:56.915455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.908419    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:20:56.915455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.908485    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:20:57.001403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.997120    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.001403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.997182    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:20:57.001403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:56.997489    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:20:57.390755 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:20:57.390705    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:20:57.414337 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:57.413870    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.414337 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:57.413942    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:20:57.928475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:57.902533    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:20:57.928475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:57.902609    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:20:58.048432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.042986    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:20:58.048432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.043051    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:20:58.437411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.428868    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2" event={"ID":"76cb7287-9039-47a4-9afa-e3f310eb7c26","Type":"ContainerDied","Data":"2fc1a08a573e3e9f1e11a359f729ee1d89aa8f329bb14e2bfc107067b13a761d"}
Nov 03 16:20:58.437411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.428934    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc1a08a573e3e9f1e11a359f729ee1d89aa8f329bb14e2bfc107067b13a761d"
Nov 03 16:20:58.473653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.473608    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:20:58.516058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.516002    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:20:58.516690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.506493    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.531663 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.531626    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:58.691471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.674633    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl\" (UniqueName: \"kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl\") pod \"76cb7287-9039-47a4-9afa-e3f310eb7c26\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") "
Nov 03 16:20:58.716413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.691838    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca\") pod \"76cb7287-9039-47a4-9afa-e3f310eb7c26\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") "
Nov 03 16:20:58.716818 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.716783    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca\") pod \"76cb7287-9039-47a4-9afa-e3f310eb7c26\" (UID: \"76cb7287-9039-47a4-9afa-e3f310eb7c26\") "
Nov 03 16:20:58.749414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.721175    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca" pod "76cb7287-9039-47a4-9afa-e3f310eb7c26" (UID: "76cb7287-9039-47a4-9afa-e3f310eb7c26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:20:58.774410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.750809    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.802417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.801559    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.827417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.826594    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:20:58.852413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.829097    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:58.852413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.848191    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:20:58.852831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.852793    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:58.909454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.909340    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.065167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.909431    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:20:59.066253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.909690    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.066253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:58.910841    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.066253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.009104    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.066579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.066539    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.118417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.097939    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca" pod "76cb7287-9039-47a4-9afa-e3f310eb7c26" (UID: "76cb7287-9039-47a4-9afa-e3f310eb7c26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:20:59.144412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.098423    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl" pod "76cb7287-9039-47a4-9afa-e3f310eb7c26" (UID: "76cb7287-9039-47a4-9afa-e3f310eb7c26"). InnerVolumeSpecName "kube-api-access-8zncl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:20:59.185796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.098495    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.186060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.186028    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:20:59.186192 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.148656    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.199180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.183269    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cb7287-9039-47a4-9afa-e3f310eb7c26-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:59.212417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.206488    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.272569 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.272524    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.376763 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.376674    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zncl\" (UniqueName: \"kubernetes.io/projected/76cb7287-9039-47a4-9afa-e3f310eb7c26-kube-api-access-8zncl\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:20:59.417810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.417777    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.419485 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.419445    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:20:59.518271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.518219    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.520266 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.520240    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:20:59.677412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.671428    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"
Nov 03 16:20:59.812572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.812522    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.839113 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.828455    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:20:59.839982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.839957    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:20:59.892563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.892527    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:20:59.904444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.867608    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.918424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.867645    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded"
Nov 03 16:20:59.926411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.849821    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:20:59.992470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.962910    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:20:59.992470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.989511    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:20:59.992470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.963520    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:20:59.992470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:20:59.989643    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:00.067413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.048323    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:21:00.067413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.048405    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:21:00.143860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.104019    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:00.143860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.129242    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:00.143860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.129967    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.143860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.130216    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:00.227446 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.186486    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.227446 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.186886    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:00.287498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.230465    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.287498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.230880    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:00.431448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.427117    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:21:00.431448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.427181    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:21:00.516047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.516016    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:00.535597 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.522079    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.680454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.675482    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:00.807764 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.807696    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:00.849237 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.849180    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.098060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.849527    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.101591 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:00.929341    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.106416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.032790    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.157429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.088529    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.157429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.136614    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:21:01.157429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.123167    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.157429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.090812    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.157429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.094182    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.157429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.137369    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:21:01.285424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.258007    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:01.285424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.258099    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:21:01.285424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.258150    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:01.285424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.283259    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.286137 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.286095    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.390679 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.390498    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:01.394413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.391169    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.414881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.414835    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.415159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.415124    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:21:01.525760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.525731    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:01.531914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.531871    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.538094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.538053    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:01.613410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.605534    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.690792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.690742    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:01.696311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.696280    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:01.867420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.856645    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:01.893441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.882202    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:01.905334 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.905301    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:21:01.905775 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.905745    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:21:01.929553 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.908147    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:01.978647 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:01.978497    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:02.018865 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.018824    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.019142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.019111    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:21:02.288016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.287913    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:02.289309 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.289274    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:02.373963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.373927    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:02.410220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.410182    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.410552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.410504    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:21:02.602846 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:02.602753    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:21:02.902200 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.902112    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.902425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.902220    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:21:02.993338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.993303    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:21:02.993729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:02.993678    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:21:03.649726 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:03.647775    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:21:03.649726 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:03.648431    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:21:03.658354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:03.658240    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:21:03.793423 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:03.791308    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:03.793423 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:03.791611    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:04.275833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.275792    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:21:04.276932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.276901    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:21:04.277928 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.277903    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:21:04.537480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.536814    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:21:04.541434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.538180    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:21:04.542413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.542031    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:21:04.725668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.725634    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"]
Nov 03 16:21:04.726354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.726336    4964 cpu_manager.go:414] "containerMap: removing container" podUID="76cb7287-9039-47a4-9afa-e3f310eb7c26" containerName="revoke-keys"
Nov 03 16:21:04.726458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.726355    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cb7287-9039-47a4-9afa-e3f310eb7c26" containerName="revoke-keys"
Nov 03 16:21:04.726524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.726478    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cb7287-9039-47a4-9afa-e3f310eb7c26" containerName="revoke-keys"
Nov 03 16:21:04.731601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.731575    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.739793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.739657    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:04.739793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.739757    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:04.751241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.751204    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"]
Nov 03 16:21:04.819224 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:04.819131    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:04.851773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.851732    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.851773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.851781    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.851999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.851959    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2r2\" (UniqueName: \"kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.953035 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.952997    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.953259 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.953054    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.953259 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.953224    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2r2\" (UniqueName: \"kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.953870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.953842    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.955815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.955788    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:04.965658 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:04.965623    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2r2\" (UniqueName: \"kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2\") pod \"maas-api-revoke-sub-39f9fd18-fbcxb\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:05.060256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:05.060218    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:05.357084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:05.356972    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"]
Nov 03 16:21:05.366742 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:21:05.366712    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6caf06c_371a_4c97_b999_01a057bf3f8b.slice/crio-bf14f62c0d26165e6e47c95a940406bf943fbf9058f14c2e838e665ef8d6a87f WatchSource:0}: Error finding container bf14f62c0d26165e6e47c95a940406bf943fbf9058f14c2e838e665ef8d6a87f: Status 404 returned error can't find the container with id bf14f62c0d26165e6e47c95a940406bf943fbf9058f14c2e838e665ef8d6a87f
Nov 03 16:21:05.613625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:05.613540    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb" event={"ID":"d6caf06c-371a-4c97-b999-01a057bf3f8b","Type":"ContainerStarted","Data":"bf14f62c0d26165e6e47c95a940406bf943fbf9058f14c2e838e665ef8d6a87f"}
Nov 03 16:21:06.626857 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:06.626770    4964 generic.go:356] "Generic (PLEG): container finished" podID="d6caf06c-371a-4c97-b999-01a057bf3f8b" containerID="0edbccd58f3b8f8cef271e030d07b81c1e0e303b24dbabd659533bc200004153" exitCode=0
Nov 03 16:21:06.627265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:06.626858    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb" event={"ID":"d6caf06c-371a-4c97-b999-01a057bf3f8b","Type":"ContainerDied","Data":"0edbccd58f3b8f8cef271e030d07b81c1e0e303b24dbabd659533bc200004153"}
Nov 03 16:21:07.409720 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:07.409682    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:07.836177 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.836134    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:07.909212 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.909168    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca\") pod \"d6caf06c-371a-4c97-b999-01a057bf3f8b\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") "
Nov 03 16:21:07.909421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.909228    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2\" (UniqueName: \"kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2\") pod \"d6caf06c-371a-4c97-b999-01a057bf3f8b\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") "
Nov 03 16:21:07.909519 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.909497    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca\") pod \"d6caf06c-371a-4c97-b999-01a057bf3f8b\" (UID: \"d6caf06c-371a-4c97-b999-01a057bf3f8b\") "
Nov 03 16:21:07.909941 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.909911    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca" pod "d6caf06c-371a-4c97-b999-01a057bf3f8b" (UID: "d6caf06c-371a-4c97-b999-01a057bf3f8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:07.910201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.910178    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:07.911863 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.911832    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2" pod "d6caf06c-371a-4c97-b999-01a057bf3f8b" (UID: "d6caf06c-371a-4c97-b999-01a057bf3f8b"). InnerVolumeSpecName "kube-api-access-jg2r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:07.913306 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.913277    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca" pod "d6caf06c-371a-4c97-b999-01a057bf3f8b" (UID: "d6caf06c-371a-4c97-b999-01a057bf3f8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:07.952717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:07.952620    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:21:08.011431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:08.011349    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6caf06c-371a-4c97-b999-01a057bf3f8b-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:08.011431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:08.011382    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2r2\" (UniqueName: \"kubernetes.io/projected/d6caf06c-371a-4c97-b999-01a057bf3f8b-kube-api-access-jg2r2\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:08.645243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:08.645210    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"
Nov 03 16:21:08.645470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:08.645200    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb" event={"ID":"d6caf06c-371a-4c97-b999-01a057bf3f8b","Type":"ContainerDied","Data":"bf14f62c0d26165e6e47c95a940406bf943fbf9058f14c2e838e665ef8d6a87f"}
Nov 03 16:21:08.645470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:08.645346    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf14f62c0d26165e6e47c95a940406bf943fbf9058f14c2e838e665ef8d6a87f"
Nov 03 16:21:10.157405 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.156997    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"]
Nov 03 16:21:10.165408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.158134    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d6caf06c-371a-4c97-b999-01a057bf3f8b" containerName="revoke-keys"
Nov 03 16:21:10.165408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.158162    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6caf06c-371a-4c97-b999-01a057bf3f8b" containerName="revoke-keys"
Nov 03 16:21:10.165408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.158330    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6caf06c-371a-4c97-b999-01a057bf3f8b" containerName="revoke-keys"
Nov 03 16:21:10.202414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.202037    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.202414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.202259    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"]
Nov 03 16:21:10.210530 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.210344    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:10.225963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.213572    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:10.288233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.288185    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.419439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.419316    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlsx\" (UniqueName: \"kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.435426 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.419738    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.436639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.436598    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.588258 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.588210    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.637097 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.637047    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlsx\" (UniqueName: \"kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.638179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.638144    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.647253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.637584    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:10.678116 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.678030    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.725448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.724912    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.742422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.726186    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlsx\" (UniqueName: \"kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx\") pod \"maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.779959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.779635    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:10.846429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.845901    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.869427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.868024    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.926997 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.919929    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.938094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.937999    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:10.992411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:10.992347    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.125411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.124920    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.166561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.166509    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.258470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.254491    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.292612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.292546    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:21:11.293980 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.293955    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:11.467415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.458695    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.493421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.461787    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.648590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.648479    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.690751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.650352    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.833310    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.916179    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.916469    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.961164    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.961226    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.923659    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:11.975410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.962863    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:11.998936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.998889    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.000316 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:11.999413    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:12.194411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.127205    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:12.224015 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.223963    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.249466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.248517    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Nov 03 16:21:12.253444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.250453    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.253444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.251882    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:12.287107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.287065    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.289404 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.289354    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:12.291656 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.291623    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.293210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.293166    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.294585 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.294565    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:12.307603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.307554    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.557563 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.514866    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:21:12.584000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.582540    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:12.584000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.583876    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:12.585425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.584538    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:12.681412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:12.649033    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.061552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.061494    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.166419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.061809    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.166419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.061888    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.166419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.061916    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.166419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.061958    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.321468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.319136    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.422420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.414776    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.570414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.524498    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.570414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.524897    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.570414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.525162    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.613061 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.612951    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:13.654454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.653638    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:13.962881 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:13.959230    4964 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.339s"
Nov 03 16:21:13.962881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.959352    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:13.962881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.959464    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:21:13.962881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:13.959616    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:14.114957 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.102121    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"]
Nov 03 16:21:14.199412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.184593    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.199412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.184990    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.199412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.185315    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.199412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.185633    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.305325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.304470    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k" event={"ID":"c89826e1-65c6-4480-9eab-e88f239d5ec9","Type":"ContainerStarted","Data":"9d953acdd3a428a7d2b3090ecbf7a38bf59d2ea33bea292106557e8bcc9e2a7c"}
Nov 03 16:21:14.323571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.323047    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.323571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.323455    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Nov 03 16:21:14.436463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.434537    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.574710 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.574545    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.576161 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.576129    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.577427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.577404    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.817412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.816966    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:14.842463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.839336    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:14.881420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:14.869011    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 03 16:21:15.335412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.334374    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:15.356976 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.356903    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:15.376948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.358120    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:15.376948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.358461    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:15.376948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.358649    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:15.496919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.494161    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:15.597092 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.596963    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:15.604077 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.603525    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:21:15.652559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.651926    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:15.819705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.818577    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:15.840555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:15.839500    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 16:21:16.022930 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.021783    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:16.028218 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.027351    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:16.033420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.031086    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:16.036481 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.036401    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Nov 03 16:21:16.046014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.045370    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:16.047565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.047533    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:21:16.085796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.082913    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"]
Nov 03 16:21:16.085796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.083013    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:16.090867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.090842    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:16.110132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.110099    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:16.112988 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.112957    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:16.126454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.126425    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:16.126638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.126568    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:16.718471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.718244    4964 generic.go:356] "Generic (PLEG): container finished" podID="c89826e1-65c6-4480-9eab-e88f239d5ec9" containerID="c82f12c6e28d02d62f42fab4e5917bab0b4eb08a2b1c086d23a2babf03e069dd" exitCode=0
Nov 03 16:21:16.720119 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:16.719554    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k" event={"ID":"c89826e1-65c6-4480-9eab-e88f239d5ec9","Type":"ContainerDied","Data":"c82f12c6e28d02d62f42fab4e5917bab0b4eb08a2b1c086d23a2babf03e069dd"}
Nov 03 16:21:17.427235 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:17.427186    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8192ee_f489_4aff_9b92_a4b476f61ca1.slice/crio-39721a9048b4beeee727aa299bc3839fa3578d86d193f71472b2b937962ffa34\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:17.678174 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.678091    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"]
Nov 03 16:21:17.682891 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.682865    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.692513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.692479    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"]
Nov 03 16:21:17.762613 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.762575    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.763111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.763083    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbkl\" (UniqueName: \"kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.763249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.763218    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.868860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.866810    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.868860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.867121    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.868860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.867191    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbkl\" (UniqueName: \"kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.868860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.868442    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.880014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.879975    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.903606 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.902243    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbkl\" (UniqueName: \"kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl\") pod \"maas-api-revoke-sub-2c89d8ba-cccn2\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:17.952239 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:17.952161    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:18.003129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.003085    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:18.069935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.069899    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx\" (UniqueName: \"kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx\") pod \"c89826e1-65c6-4480-9eab-e88f239d5ec9\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") "
Nov 03 16:21:18.070087 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.069992    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca\") pod \"c89826e1-65c6-4480-9eab-e88f239d5ec9\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") "
Nov 03 16:21:18.070172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.070105    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca\") pod \"c89826e1-65c6-4480-9eab-e88f239d5ec9\" (UID: \"c89826e1-65c6-4480-9eab-e88f239d5ec9\") "
Nov 03 16:21:18.070445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.070409    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca" pod "c89826e1-65c6-4480-9eab-e88f239d5ec9" (UID: "c89826e1-65c6-4480-9eab-e88f239d5ec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:18.070946 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.070920    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:18.073231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.073183    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca" pod "c89826e1-65c6-4480-9eab-e88f239d5ec9" (UID: "c89826e1-65c6-4480-9eab-e88f239d5ec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:18.073330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.073261    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx" pod "c89826e1-65c6-4480-9eab-e88f239d5ec9" (UID: "c89826e1-65c6-4480-9eab-e88f239d5ec9"). InnerVolumeSpecName "kube-api-access-tjlsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:18.172440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.172407    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjlsx\" (UniqueName: \"kubernetes.io/projected/c89826e1-65c6-4480-9eab-e88f239d5ec9-kube-api-access-tjlsx\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:18.172440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.172438    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89826e1-65c6-4480-9eab-e88f239d5ec9-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:18.305052 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.304961    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"]
Nov 03 16:21:18.305052 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:21:18.305029    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e0921b_e10b_44d0_833a_c52031d5e6d7.slice/crio-5c9b3b0990d09e83ceafd4c5a48c91216ee337001dbe1a54a71acbadb57c127a WatchSource:0}: Error finding container 5c9b3b0990d09e83ceafd4c5a48c91216ee337001dbe1a54a71acbadb57c127a: Status 404 returned error can't find the container with id 5c9b3b0990d09e83ceafd4c5a48c91216ee337001dbe1a54a71acbadb57c127a
Nov 03 16:21:18.738627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.738598    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"
Nov 03 16:21:18.738627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.738609    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k" event={"ID":"c89826e1-65c6-4480-9eab-e88f239d5ec9","Type":"ContainerDied","Data":"9d953acdd3a428a7d2b3090ecbf7a38bf59d2ea33bea292106557e8bcc9e2a7c"}
Nov 03 16:21:18.738877 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.738670    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d953acdd3a428a7d2b3090ecbf7a38bf59d2ea33bea292106557e8bcc9e2a7c"
Nov 03 16:21:18.740696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.740663    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2" event={"ID":"32e0921b-e10b-44d0-833a-c52031d5e6d7","Type":"ContainerStarted","Data":"8b0e1a9f2b5dbfa9173e24485eaa61a35a01b08a1258f53f55b21ef2e0510ca0"}
Nov 03 16:21:18.740844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.740723    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2" event={"ID":"32e0921b-e10b-44d0-833a-c52031d5e6d7","Type":"ContainerStarted","Data":"5c9b3b0990d09e83ceafd4c5a48c91216ee337001dbe1a54a71acbadb57c127a"}
Nov 03 16:21:18.763398 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.763336    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2" podStartSLOduration=1.763317042 podStartE2EDuration="1.763317042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:18.760003811 +0000 UTC m=+1140.273385586" watchObservedRunningTime="2026-11-03 16:21:18.763317042 +0000 UTC m=+1140.276698814"
Nov 03 16:21:18.838297 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.838259    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"]
Nov 03 16:21:18.839638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.839376    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c89826e1-65c6-4480-9eab-e88f239d5ec9" containerName="revoke-keys"
Nov 03 16:21:18.839638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.839416    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89826e1-65c6-4480-9eab-e88f239d5ec9" containerName="revoke-keys"
Nov 03 16:21:18.839638 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.839635    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89826e1-65c6-4480-9eab-e88f239d5ec9" containerName="revoke-keys"
Nov 03 16:21:18.859891 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.844414    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:18.859891 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.856499    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"]
Nov 03 16:21:18.988011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.987966    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:18.988168 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.988038    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2b5v\" (UniqueName: \"kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:18.988223 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:18.988161    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.089613 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.089568    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.089815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.089643    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2b5v\" (UniqueName: \"kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.089815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.089749    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.090239 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.090208    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.092091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.092059    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.102651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.102619    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2b5v\" (UniqueName: \"kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v\") pod \"maas-api-revoke-sub-034e9764-kdq5f\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.179765 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.179730    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:19.479038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.479010    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"]
Nov 03 16:21:19.481708 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:21:19.481675    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5be3fb_4c8d_4567_8a6a_07eb64d13c04.slice/crio-41583d2f88e39bf314b46394678b7ee2a6319ee0eb8d92acf64c76e506cfd51c WatchSource:0}: Error finding container 41583d2f88e39bf314b46394678b7ee2a6319ee0eb8d92acf64c76e506cfd51c: Status 404 returned error can't find the container with id 41583d2f88e39bf314b46394678b7ee2a6319ee0eb8d92acf64c76e506cfd51c
Nov 03 16:21:19.752715 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.752615    4964 generic.go:356] "Generic (PLEG): container finished" podID="32e0921b-e10b-44d0-833a-c52031d5e6d7" containerID="8b0e1a9f2b5dbfa9173e24485eaa61a35a01b08a1258f53f55b21ef2e0510ca0" exitCode=0
Nov 03 16:21:19.752715 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.752703    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2" event={"ID":"32e0921b-e10b-44d0-833a-c52031d5e6d7","Type":"ContainerDied","Data":"8b0e1a9f2b5dbfa9173e24485eaa61a35a01b08a1258f53f55b21ef2e0510ca0"}
Nov 03 16:21:19.754370 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:19.754340    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f" event={"ID":"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04","Type":"ContainerStarted","Data":"41583d2f88e39bf314b46394678b7ee2a6319ee0eb8d92acf64c76e506cfd51c"}
Nov 03 16:21:20.394347 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:20.394310    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"]
Nov 03 16:21:20.814410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:20.810027    4964 generic.go:356] "Generic (PLEG): container finished" podID="5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" containerID="dfcbd3d44b228ee44c3ecd6ff53f6a065ef67e7be903a4487dc0609e94e14258" exitCode=0
Nov 03 16:21:20.814410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:20.810806    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f" event={"ID":"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04","Type":"ContainerDied","Data":"dfcbd3d44b228ee44c3ecd6ff53f6a065ef67e7be903a4487dc0609e94e14258"}
Nov 03 16:21:21.095140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.095110    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:21.249835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.249792    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca\") pod \"32e0921b-e10b-44d0-833a-c52031d5e6d7\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") "
Nov 03 16:21:21.250057 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.249999    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca\") pod \"32e0921b-e10b-44d0-833a-c52031d5e6d7\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") "
Nov 03 16:21:21.250124 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.250100    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl\" (UniqueName: \"kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl\") pod \"32e0921b-e10b-44d0-833a-c52031d5e6d7\" (UID: \"32e0921b-e10b-44d0-833a-c52031d5e6d7\") "
Nov 03 16:21:21.250476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.250449    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca" pod "32e0921b-e10b-44d0-833a-c52031d5e6d7" (UID: "32e0921b-e10b-44d0-833a-c52031d5e6d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:21.250746 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.250729    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:21.252185 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.252148    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca" pod "32e0921b-e10b-44d0-833a-c52031d5e6d7" (UID: "32e0921b-e10b-44d0-833a-c52031d5e6d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:21.252297 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.252276    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl" pod "32e0921b-e10b-44d0-833a-c52031d5e6d7" (UID: "32e0921b-e10b-44d0-833a-c52031d5e6d7"). InnerVolumeSpecName "kube-api-access-vtbkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:21.352293 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.352190    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtbkl\" (UniqueName: \"kubernetes.io/projected/32e0921b-e10b-44d0-833a-c52031d5e6d7-kube-api-access-vtbkl\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:21.352293 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.352228    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e0921b-e10b-44d0-833a-c52031d5e6d7-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:21.835713 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.835668    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"
Nov 03 16:21:21.836512 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.836474    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2" event={"ID":"32e0921b-e10b-44d0-833a-c52031d5e6d7","Type":"ContainerDied","Data":"5c9b3b0990d09e83ceafd4c5a48c91216ee337001dbe1a54a71acbadb57c127a"}
Nov 03 16:21:21.836626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:21.836543    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c9b3b0990d09e83ceafd4c5a48c91216ee337001dbe1a54a71acbadb57c127a"
Nov 03 16:21:22.012464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.012437    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:22.169554 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.169518    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v\" (UniqueName: \"kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v\") pod \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") "
Nov 03 16:21:22.169841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.169819    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca\") pod \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") "
Nov 03 16:21:22.170006 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.169947    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca\") pod \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\" (UID: \"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04\") "
Nov 03 16:21:22.172084 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.172050    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v" pod "5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" (UID: "5d5be3fb-4c8d-4567-8a6a-07eb64d13c04"). InnerVolumeSpecName "kube-api-access-k2b5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:22.172418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.172359    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca" pod "5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" (UID: "5d5be3fb-4c8d-4567-8a6a-07eb64d13c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:22.172607 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.172579    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca" pod "5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" (UID: "5d5be3fb-4c8d-4567-8a6a-07eb64d13c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:22.174140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.174111    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:22.174242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.174145    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:22.174242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.174161    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2b5v\" (UniqueName: \"kubernetes.io/projected/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04-kube-api-access-k2b5v\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:22.855896 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.855865    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"
Nov 03 16:21:22.855896 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.855880    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f" event={"ID":"5d5be3fb-4c8d-4567-8a6a-07eb64d13c04","Type":"ContainerDied","Data":"41583d2f88e39bf314b46394678b7ee2a6319ee0eb8d92acf64c76e506cfd51c"}
Nov 03 16:21:22.856299 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:22.855923    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41583d2f88e39bf314b46394678b7ee2a6319ee0eb8d92acf64c76e506cfd51c"
Nov 03 16:21:23.996861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.996824    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"]
Nov 03 16:21:23.997870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.997848    4964 cpu_manager.go:414] "containerMap: removing container" podUID="32e0921b-e10b-44d0-833a-c52031d5e6d7" containerName="revoke-keys"
Nov 03 16:21:23.997913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.997875    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e0921b-e10b-44d0-833a-c52031d5e6d7" containerName="revoke-keys"
Nov 03 16:21:23.997913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.997892    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" containerName="revoke-keys"
Nov 03 16:21:23.997913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.997900    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" containerName="revoke-keys"
Nov 03 16:21:23.998121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.998108    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e0921b-e10b-44d0-833a-c52031d5e6d7" containerName="revoke-keys"
Nov 03 16:21:23.998154 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:23.998125    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" containerName="revoke-keys"
Nov 03 16:21:24.008000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.007968    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.035241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.035186    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"]
Nov 03 16:21:24.116833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.116781    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.116950    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117124    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117176    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117259 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117236    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4p4g\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117341 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117326    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117412    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117471    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.117553 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.117527    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.218837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.218789    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.218879    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219008    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219045    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219069    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4p4g\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219102    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219137    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219192    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219246    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219635    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219669    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219757    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.219904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.219885    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.220186 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.220170    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.221470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.221439    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.222218 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.222195    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.235273 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.235229    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.235483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.235362    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4p4g\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.339687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.339174    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"]
Nov 03 16:21:24.345330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.345295    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:24.345915 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.345888    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.360280 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.360246    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"]
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523351    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523459    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523650    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523694    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2n4\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523778    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523841    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523967    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.523995    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.533157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.524024    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.624998    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625043    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625170    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625291    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625357    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625525    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625562    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2n4\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625581    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625601    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625622    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625654    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625866 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625719    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.625866 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.625841    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.626464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.626441    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.628349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.628318    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.628669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.628649    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.638540 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.638504    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2n4\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.638889 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.638864    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.666216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.666183    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"]
Nov 03 16:21:24.666881 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:21:24.666853    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01b1dbb_80b6_4563_9cd4_d9c3583d45e7.slice/crio-b490bc0b11e73544ab9157930504ac7461ce291e06543311387fb514a76783aa WatchSource:0}: Error finding container b490bc0b11e73544ab9157930504ac7461ce291e06543311387fb514a76783aa: Status 404 returned error can't find the container with id b490bc0b11e73544ab9157930504ac7461ce291e06543311387fb514a76783aa
Nov 03 16:21:24.669915 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.669877    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:24.670009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.669964    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:24.670080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.670010    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:24.708423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.708348    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:24.883403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.883078    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" event={"ID":"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7","Type":"ContainerStarted","Data":"8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c"}
Nov 03 16:21:24.883403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.883132    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" event={"ID":"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7","Type":"ContainerStarted","Data":"b490bc0b11e73544ab9157930504ac7461ce291e06543311387fb514a76783aa"}
Nov 03 16:21:24.920064 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.920008    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"]
Nov 03 16:21:24.921115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.921035    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podStartSLOduration=1.921015286 podStartE2EDuration="1.921015286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:24.916441702 +0000 UTC m=+1146.429823478" watchObservedRunningTime="2026-11-03 16:21:24.921015286 +0000 UTC m=+1146.434397077"
Nov 03 16:21:24.925615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.925585    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.952456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.951355    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"]
Nov 03 16:21:24.978612 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.978565    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.978916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.978888    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979053    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8rj\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979240 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979200    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979324 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979247    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979324 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979295    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979354    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979466    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:24.979819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:24.979788    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.019473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.019435    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"]
Nov 03 16:21:25.023733 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.023690    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:25.023867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.023797    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:25.023867 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.023841    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:25.081773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.081738    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.081983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.081878    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082000    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082040    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082079    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8rj\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082154    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082185    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082277 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082229    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082285    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082652    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.082885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.082859    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.083167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.083148    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.083445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.083424    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.083661 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.083642    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.085761 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.085740    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.085958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.085934    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.095966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.095930    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.096144 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.096037    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8rj\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.247462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.245782    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:25.351883 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.348736    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:25.358118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.358076    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:25.358698 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.358662    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:25.394857 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.394658    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerName="payload-processing" containerID="cri-o://d5d08eba354561e5ecf2404158cedafbf3daeed4b919b1cbe093f8dd2c9f49a5" gracePeriod=25
Nov 03 16:21:25.674844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.674810    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"]
Nov 03 16:21:25.687014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.686966    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:25.687267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.687240    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:25.687370 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.687294    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:25.986644 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.986606    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" event={"ID":"577d79ca-7173-4aee-9c28-88dd5300275a","Type":"ContainerStarted","Data":"22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb"}
Nov 03 16:21:25.988020 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.987987    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" event={"ID":"577d79ca-7173-4aee-9c28-88dd5300275a","Type":"ContainerStarted","Data":"c54f7a06d7ca02d4b9bbb35aa9cbf5ab46b6bab7882c382bb05346ea915b053d"}
Nov 03 16:21:25.991602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.991547    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" event={"ID":"1d4d726f-ef35-4781-8c27-431dad4ab68a","Type":"ContainerStarted","Data":"9403a1f5f6316265e3c78d8ec04c788a04839f6bef2c82bd7b0766031a1b8a84"}
Nov 03 16:21:25.991740 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:25.991626    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" event={"ID":"1d4d726f-ef35-4781-8c27-431dad4ab68a","Type":"ContainerStarted","Data":"6fd42d41ad633b4a9e55b0b176f0d5db36f1d109a7917d0ed326ebf7f3d0856e"}
Nov 03 16:21:26.056700 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.056507    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podStartSLOduration=2.056481138 podStartE2EDuration="2.056481138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:26.035922953 +0000 UTC m=+1147.549304731" watchObservedRunningTime="2026-11-03 16:21:26.056481138 +0000 UTC m=+1147.569862909"
Nov 03 16:21:26.089657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.089577    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podStartSLOduration=2.08955155 podStartE2EDuration="2.08955155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:26.082808365 +0000 UTC m=+1147.596190142" watchObservedRunningTime="2026-11-03 16:21:26.08955155 +0000 UTC m=+1147.602933331"
Nov 03 16:21:26.253410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.252340    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:26.253410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.252539    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:26.253410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.252608    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:26.352030 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.351929    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:26.352310 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.352276    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:26.487476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.487438    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"]
Nov 03 16:21:26.494456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.494424    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"]
Nov 03 16:21:26.494833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.494763    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.500172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.500144    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:21:26.500371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.500359    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:21:26.584007 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.583952    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.584457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.584426    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.584739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.584721    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8t6x\" (UniqueName: \"kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.595395 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.594433    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"]
Nov 03 16:21:26.687705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.687656    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8t6x\" (UniqueName: \"kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.688066 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.688040    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.688459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.688436    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.694687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.694631    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.701637 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.701597    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:26.805088 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.804707    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:26.805088 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.804780    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:26.890272 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.890236    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:26.890558 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.890538    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"]
Nov 03 16:21:26.890687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.890674    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"]
Nov 03 16:21:26.961626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:26.961511    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"]
Nov 03 16:21:27.047411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.044415    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8t6x\" (UniqueName: \"kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x\") pod \"maas-api-revoke-sub-6d65a089-4wgrh\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:27.197228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.197196    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"]
Nov 03 16:21:27.229486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.222724    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.251425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.234322    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"]
Nov 03 16:21:27.260078 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.256435    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.283406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.282687    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.283406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.282796    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62qj\" (UniqueName: \"kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.310437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.284613    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:27.310437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.286702    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:27.310437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.286752    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:27.450441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.406405    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:27.450441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.406464    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:27.450441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.406683    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.450441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.406784    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.450441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.406838    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62qj\" (UniqueName: \"kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.558104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.536157    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.597237 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.595463    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62qj\" (UniqueName: \"kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.600224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.598066    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:27.708803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.708757    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93e274f6-p7vg9\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:27.759552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.715828    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:27.759966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.759838    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:27.901874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:27.901833    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:28.184373 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.184323    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:28.185155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.185107    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:28.186156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.186130    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:28.205897 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.203192    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:28.297498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.293843    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:28.297498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.294192    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:28.386664 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.363334    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:28.386664 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.363444    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:28.536411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.535550    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:28.683600 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.598212    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:28.683600 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.598951    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:28.683600 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.622703    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 16:21:28.683600 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.669476    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:28.749697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.733054    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:28.749697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.733120    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:28.997580 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:28.989210    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.061466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.059262    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"]
Nov 03 16:21:29.061466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.059353    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:29.196436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.192947    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.199694 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.199654    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:29.231960 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.231916    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.232979 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.232939    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.233958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.233938    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:29.234157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.234134    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:29.274817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.272852    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:29.274817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.272908    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:29.352675 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.343238    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.391075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.389784    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:29.391075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.389841    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:29.462435 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.461895    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.650684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.650578    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.651951 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.651931    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:29.670966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.670921    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.671900 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.671875    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:29.680210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.680172    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:29.722532 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.699512    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:29.722532 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.699578    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"]
Nov 03 16:21:29.958425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.945040    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:29.977878 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:29.977840    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.047064 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.047016    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:30.047707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.017663    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh" event={"ID":"dbba35b9-e74c-4547-857a-16445bc1bd55","Type":"ContainerStarted","Data":"d9c4cb0fc6440e8c0beeba00a1f4c9788f20f4a9f53801e579d47455ea42df89"}
Nov 03 16:21:30.059736 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.059692    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded"
Nov 03 16:21:30.064071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.064034    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9" event={"ID":"126c9c1e-9b01-427a-9b3e-878880651330","Type":"ContainerStarted","Data":"81e5743407e322d5b2168c437020e1f142b2be75283c1eccc0740edb437a16c9"}
Nov 03 16:21:30.168577 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.168475    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.203880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.203824    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:30.204601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.204559    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.276465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.267261    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.276465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.267319    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:30.276465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.267430    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.337463 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.326159    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:30.360997 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.356195    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.360997 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.356255    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:30.472574 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.472435    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.507412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.506914    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:30.698460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.652586    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.734412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.731291    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:30.734412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.731691    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:30.734412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.731734    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:30.897410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:30.893781    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:31.001126 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.001087    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:31.021822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.021789    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:31.022076 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.022063    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:31.196578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.192239    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:31.226467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.223435    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:31.249410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.230179    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:31.560937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.560844    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:31.561546 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.561513    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh" event={"ID":"dbba35b9-e74c-4547-857a-16445bc1bd55","Type":"ContainerStarted","Data":"e693efab1781a5012cce97b54710d282842f54d062901dc37a9efe33e22930bf"}
Nov 03 16:21:31.561732 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.561710    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9" event={"ID":"126c9c1e-9b01-427a-9b3e-878880651330","Type":"ContainerStarted","Data":"483c8f27c754d4ccd4d77c1f6875e9f294fd3c5691413a3bc3d081b02bc7ad4d"}
Nov 03 16:21:31.562102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.249916    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:31.563177 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.347690    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:31.563459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.529803    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:21:31.563815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.384160    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:31.563998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.479601    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:31.564070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.564033    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:31.564070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.502917    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:31.564185 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.564098    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:31.565895 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.565873    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:31.588959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.588915    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:31.737570 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.737516    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 03 16:21:31.775419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.774354    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:31.775419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.774450    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:31.827194 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.827068    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" containerID="cri-o://cb73a208731ebc6ce29ab4f49eb861f5837d6e6792548fae5528e170db6bdae6" gracePeriod=45
Nov 03 16:21:31.993415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:31.990978    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:32.083881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.065996    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:32.159420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.143095    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:32.160070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.160043    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:32.244333 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.244297    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"]
Nov 03 16:21:32.276047 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.247303    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:32.276347 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.276312    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:32.293521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.293446    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:32.293720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.293705    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:32.294862 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.294845    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.301413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.299944    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"]
Nov 03 16:21:32.338046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.337641    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": dial tcp 10.132.0.58:8000: connect: connection refused"
Nov 03 16:21:32.346287 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.345878    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 16:21:32.346287 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.345932    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 16:21:32.481822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.481460    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.481822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.481577    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.481822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.481645    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xrr\" (UniqueName: \"kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.490178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.490125    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9" podStartSLOduration=6.490110019 podStartE2EDuration="6.490110019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:31.627772961 +0000 UTC m=+1153.141154741" watchObservedRunningTime="2026-11-03 16:21:32.490110019 +0000 UTC m=+1154.003491795"
Nov 03 16:21:32.494979 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.494928    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh" podStartSLOduration=6.494913327 podStartE2EDuration="6.494913327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:31.750832715 +0000 UTC m=+1153.264214493" watchObservedRunningTime="2026-11-03 16:21:32.494913327 +0000 UTC m=+1154.008295155"
Nov 03 16:21:32.587478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.587369    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.587478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.587482    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.588419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.587535    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xrr\" (UniqueName: \"kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.593413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.588675    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.608627 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.599913    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"]
Nov 03 16:21:32.618472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.615605    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.618472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.615857    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"]
Nov 03 16:21:32.693922 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.692620    4964 generic.go:356] "Generic (PLEG): container finished" podID="126c9c1e-9b01-427a-9b3e-878880651330" containerID="483c8f27c754d4ccd4d77c1f6875e9f294fd3c5691413a3bc3d081b02bc7ad4d" exitCode=0
Nov 03 16:21:32.693922 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.692864    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9" event={"ID":"126c9c1e-9b01-427a-9b3e-878880651330","Type":"ContainerDied","Data":"483c8f27c754d4ccd4d77c1f6875e9f294fd3c5691413a3bc3d081b02bc7ad4d"}
Nov 03 16:21:32.694990 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.694354    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xrr\" (UniqueName: \"kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr\") pod \"maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.702362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.701243    4964 generic.go:356] "Generic (PLEG): container finished" podID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerID="cb73a208731ebc6ce29ab4f49eb861f5837d6e6792548fae5528e170db6bdae6" exitCode=0
Nov 03 16:21:32.702362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.701428    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" event={"ID":"ec89f4cf-6327-4b51-9511-8800454e76c9","Type":"ContainerDied","Data":"cb73a208731ebc6ce29ab4f49eb861f5837d6e6792548fae5528e170db6bdae6"}
Nov 03 16:21:32.730813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.730306    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:32.730813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.730361    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:32.742559 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:32.742478    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:21:32.765062 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.764805    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:32.822900 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.822842    4964 generic.go:356] "Generic (PLEG): container finished" podID="dbba35b9-e74c-4547-857a-16445bc1bd55" containerID="e693efab1781a5012cce97b54710d282842f54d062901dc37a9efe33e22930bf" exitCode=0
Nov 03 16:21:32.930417 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:32.929537    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh" event={"ID":"dbba35b9-e74c-4547-857a-16445bc1bd55","Type":"ContainerDied","Data":"e693efab1781a5012cce97b54710d282842f54d062901dc37a9efe33e22930bf"}
Nov 03 16:21:33.301901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.301866    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:33.302134 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.302102    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:33.492491 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.488352    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:21:33.584074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.583985    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm\") pod \"ec89f4cf-6327-4b51-9511-8800454e76c9\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") "
Nov 03 16:21:33.584444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.584419    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home\") pod \"ec89f4cf-6327-4b51-9511-8800454e76c9\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") "
Nov 03 16:21:33.584670 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.584651    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir\") pod \"ec89f4cf-6327-4b51-9511-8800454e76c9\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") "
Nov 03 16:21:33.584983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.584861    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir" pod "ec89f4cf-6327-4b51-9511-8800454e76c9" (UID: "ec89f4cf-6327-4b51-9511-8800454e76c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:33.584983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.584891    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache\") pod \"ec89f4cf-6327-4b51-9511-8800454e76c9\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") "
Nov 03 16:21:33.584983 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.584937    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home" pod "ec89f4cf-6327-4b51-9511-8800454e76c9" (UID: "ec89f4cf-6327-4b51-9511-8800454e76c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:33.597368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.595954    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs\") pod \"ec89f4cf-6327-4b51-9511-8800454e76c9\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") "
Nov 03 16:21:33.597368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.596041    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4\" (UniqueName: \"kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4\") pod \"ec89f4cf-6327-4b51-9511-8800454e76c9\" (UID: \"ec89f4cf-6327-4b51-9511-8800454e76c9\") "
Nov 03 16:21:33.597368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.596978    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:33.597368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.597001    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:33.599976 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.599939    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache" pod "ec89f4cf-6327-4b51-9511-8800454e76c9" (UID: "ec89f4cf-6327-4b51-9511-8800454e76c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:33.701716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.701681    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4" pod "ec89f4cf-6327-4b51-9511-8800454e76c9" (UID: "ec89f4cf-6327-4b51-9511-8800454e76c9"). InnerVolumeSpecName "kube-api-access-dmgh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:33.701870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.701727    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:33.746837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.726023    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 16:21:33.746837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.726083    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 16:21:33.746837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.727199    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs" pod "ec89f4cf-6327-4b51-9511-8800454e76c9" (UID: "ec89f4cf-6327-4b51-9511-8800454e76c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:33.746837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.746551    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm" pod "ec89f4cf-6327-4b51-9511-8800454e76c9" (UID: "ec89f4cf-6327-4b51-9511-8800454e76c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:33.821771 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.817606    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec89f4cf-6327-4b51-9511-8800454e76c9-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:33.821771 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.817644    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec89f4cf-6327-4b51-9511-8800454e76c9-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:33.821771 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:33.817659    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmgh4\" (UniqueName: \"kubernetes.io/projected/ec89f4cf-6327-4b51-9511-8800454e76c9-kube-api-access-dmgh4\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.064442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.061155    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"
Nov 03 16:21:34.064442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.062235    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" event={"ID":"ec89f4cf-6327-4b51-9511-8800454e76c9","Type":"ContainerDied","Data":"4e5a4f6f4c5f6b89a34e6bc3fbac186c508baff1852bc9d9680265d4230ebe31"}
Nov 03 16:21:34.064442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.062313    4964 scope.go:122] "RemoveContainer" containerID="cb73a208731ebc6ce29ab4f49eb861f5837d6e6792548fae5528e170db6bdae6"
Nov 03 16:21:34.136270 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.135679    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"]
Nov 03 16:21:34.167269 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.167234    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"]
Nov 03 16:21:34.217361 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.217326    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5"]
Nov 03 16:21:34.246908 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.246874    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 16:21:34.247260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.247214    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 16:21:34.265407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.263811    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:34.267553 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.267518    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:34.270538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.270513    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:21:34.348170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.348133    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/premium-w0-d569fb-kserve-5b5ff7959b-hc6f5" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 16:21:34.469331 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.469289    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:34.562022 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.561999    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:34.580408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.579709    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca\") pod \"126c9c1e-9b01-427a-9b3e-878880651330\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") "
Nov 03 16:21:34.580408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.579878    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj\" (UniqueName: \"kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj\") pod \"126c9c1e-9b01-427a-9b3e-878880651330\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") "
Nov 03 16:21:34.580408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.580092    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca\") pod \"126c9c1e-9b01-427a-9b3e-878880651330\" (UID: \"126c9c1e-9b01-427a-9b3e-878880651330\") "
Nov 03 16:21:34.581019 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.580988    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca" pod "126c9c1e-9b01-427a-9b3e-878880651330" (UID: "126c9c1e-9b01-427a-9b3e-878880651330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:34.598748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.594954    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj" pod "126c9c1e-9b01-427a-9b3e-878880651330" (UID: "126c9c1e-9b01-427a-9b3e-878880651330"). InnerVolumeSpecName "kube-api-access-f62qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:34.598748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.596362    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca" pod "126c9c1e-9b01-427a-9b3e-878880651330" (UID: "126c9c1e-9b01-427a-9b3e-878880651330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:34.668140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.668103    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" path="/var/lib/kubelet/pods/ec89f4cf-6327-4b51-9511-8800454e76c9/volumes"
Nov 03 16:21:34.681777 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.681739    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x\" (UniqueName: \"kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x\") pod \"dbba35b9-e74c-4547-857a-16445bc1bd55\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") "
Nov 03 16:21:34.681957 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.681923    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca\") pod \"dbba35b9-e74c-4547-857a-16445bc1bd55\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") "
Nov 03 16:21:34.682157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.682065    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca\") pod \"dbba35b9-e74c-4547-857a-16445bc1bd55\" (UID: \"dbba35b9-e74c-4547-857a-16445bc1bd55\") "
Nov 03 16:21:34.682813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.682783    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62qj\" (UniqueName: \"kubernetes.io/projected/126c9c1e-9b01-427a-9b3e-878880651330-kube-api-access-f62qj\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.682813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.682818    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.682992 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.682835    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126c9c1e-9b01-427a-9b3e-878880651330-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.683112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.683087    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca" pod "dbba35b9-e74c-4547-857a-16445bc1bd55" (UID: "dbba35b9-e74c-4547-857a-16445bc1bd55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:34.684849 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.684824    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca" pod "dbba35b9-e74c-4547-857a-16445bc1bd55" (UID: "dbba35b9-e74c-4547-857a-16445bc1bd55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:34.687318 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.687297    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x" pod "dbba35b9-e74c-4547-857a-16445bc1bd55" (UID: "dbba35b9-e74c-4547-857a-16445bc1bd55"). InnerVolumeSpecName "kube-api-access-h8t6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:34.784899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.784854    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8t6x\" (UniqueName: \"kubernetes.io/projected/dbba35b9-e74c-4547-857a-16445bc1bd55-kube-api-access-h8t6x\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.784899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.784899    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.785158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.784918    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbba35b9-e74c-4547-857a-16445bc1bd55-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:34.972465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:34.972423    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"]
Nov 03 16:21:35.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.072703    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh" event={"ID":"dbba35b9-e74c-4547-857a-16445bc1bd55","Type":"ContainerDied","Data":"d9c4cb0fc6440e8c0beeba00a1f4c9788f20f4a9f53801e579d47455ea42df89"}
Nov 03 16:21:35.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.072762    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c4cb0fc6440e8c0beeba00a1f4c9788f20f4a9f53801e579d47455ea42df89"
Nov 03 16:21:35.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.072827    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"
Nov 03 16:21:35.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.076424    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn" event={"ID":"d754ffb0-1a73-4596-b18f-d7fdbbcccca6","Type":"ContainerStarted","Data":"2da81af0724ab5b95187a68830d68688e2027a4d305d68994f5b855fe629f88c"}
Nov 03 16:21:35.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.076484    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn" event={"ID":"d754ffb0-1a73-4596-b18f-d7fdbbcccca6","Type":"ContainerStarted","Data":"5f096bca8d2eced9b9c60605438a7899a0655ed07c9281d07aa35cb8a8883785"}
Nov 03 16:21:35.080970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.080665    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"
Nov 03 16:21:35.081572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.081271    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9" event={"ID":"126c9c1e-9b01-427a-9b3e-878880651330","Type":"ContainerDied","Data":"81e5743407e322d5b2168c437020e1f142b2be75283c1eccc0740edb437a16c9"}
Nov 03 16:21:35.081572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.081314    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e5743407e322d5b2168c437020e1f142b2be75283c1eccc0740edb437a16c9"
Nov 03 16:21:35.098211 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.098152    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn" podStartSLOduration=5.098131983 podStartE2EDuration="5.098131983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:35.095713994 +0000 UTC m=+1156.609095764" watchObservedRunningTime="2026-11-03 16:21:35.098131983 +0000 UTC m=+1156.611513759"
Nov 03 16:21:35.680456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680123    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"]
Nov 03 16:21:35.680948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680801    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main"
Nov 03 16:21:35.680948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680815    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main"
Nov 03 16:21:35.680948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680841    4964 cpu_manager.go:414] "containerMap: removing container" podUID="126c9c1e-9b01-427a-9b3e-878880651330" containerName="revoke-keys"
Nov 03 16:21:35.680948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680849    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126c9c1e-9b01-427a-9b3e-878880651330" containerName="revoke-keys"
Nov 03 16:21:35.680948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680860    4964 cpu_manager.go:414] "containerMap: removing container" podUID="dbba35b9-e74c-4547-857a-16445bc1bd55" containerName="revoke-keys"
Nov 03 16:21:35.680948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680865    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbba35b9-e74c-4547-857a-16445bc1bd55" containerName="revoke-keys"
Nov 03 16:21:35.681296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680982    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="126c9c1e-9b01-427a-9b3e-878880651330" containerName="revoke-keys"
Nov 03 16:21:35.681296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.680992    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbba35b9-e74c-4547-857a-16445bc1bd55" containerName="revoke-keys"
Nov 03 16:21:35.681296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.681003    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec89f4cf-6327-4b51-9511-8800454e76c9" containerName="main"
Nov 03 16:21:35.684594 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.684575    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.744648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.726220    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"]
Nov 03 16:21:35.767817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.767666    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:35.768242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.768211    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:35.802444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.802245    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr86l\" (UniqueName: \"kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.802444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.802324    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.812913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.806641    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.877711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.871889    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:35.877711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.874692    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:35.880716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.880688    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:35.917315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.912398    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr86l\" (UniqueName: \"kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.917315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.913323    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.917315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.914542    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.917315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.916655    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.932847 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.932712    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:35.944147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:35.937651    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr86l\" (UniqueName: \"kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l\") pod \"maas-api-revoke-sub-63aadd61-k6t29\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:36.003413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.002871    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:36.185775 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.185696    4964 generic.go:356] "Generic (PLEG): container finished" podID="d754ffb0-1a73-4596-b18f-d7fdbbcccca6" containerID="2da81af0724ab5b95187a68830d68688e2027a4d305d68994f5b855fe629f88c" exitCode=0
Nov 03 16:21:36.188345 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.187498    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn" event={"ID":"d754ffb0-1a73-4596-b18f-d7fdbbcccca6","Type":"ContainerDied","Data":"2da81af0724ab5b95187a68830d68688e2027a4d305d68994f5b855fe629f88c"}
Nov 03 16:21:36.250882 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.249606    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:36.252839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.252414    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:36.266797 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.266764    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:21:36.471781 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:21:36.471750    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2f81d1_ac41_440e_96f1_bcc075ea7ae4.slice/crio-00b233bbec7f20b6c749f869e30e6726eea8b424da80aa4e1c0fc0c97e0464a0 WatchSource:0}: Error finding container 00b233bbec7f20b6c749f869e30e6726eea8b424da80aa4e1c0fc0c97e0464a0: Status 404 returned error can't find the container with id 00b233bbec7f20b6c749f869e30e6726eea8b424da80aa4e1c0fc0c97e0464a0
Nov 03 16:21:36.472917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:36.472889    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"]
Nov 03 16:21:37.215266 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.209128    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29" event={"ID":"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4","Type":"ContainerStarted","Data":"15a02e25a714a67de4550afbfecc4d5727952db880596186d6115dc8f1fe2c19"}
Nov 03 16:21:37.215266 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.209206    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29" event={"ID":"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4","Type":"ContainerStarted","Data":"00b233bbec7f20b6c749f869e30e6726eea8b424da80aa4e1c0fc0c97e0464a0"}
Nov 03 16:21:37.246434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.241234    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29" podStartSLOduration=2.241211198 podStartE2EDuration="2.241211198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:37.234966433 +0000 UTC m=+1158.748348210" watchObservedRunningTime="2026-11-03 16:21:37.241211198 +0000 UTC m=+1158.754592966"
Nov 03 16:21:37.374659 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.374125    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"]
Nov 03 16:21:37.385267 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.385170    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d569fb-29j5k"]
Nov 03 16:21:37.461019 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.460991    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:37.573611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.573574    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr\" (UniqueName: \"kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr\") pod \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") "
Nov 03 16:21:37.573760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.573730    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca\") pod \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") "
Nov 03 16:21:37.573893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.573859    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca\") pod \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\" (UID: \"d754ffb0-1a73-4596-b18f-d7fdbbcccca6\") "
Nov 03 16:21:37.574830 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.574798    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca" pod "d754ffb0-1a73-4596-b18f-d7fdbbcccca6" (UID: "d754ffb0-1a73-4596-b18f-d7fdbbcccca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:37.576780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.576755    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca" pod "d754ffb0-1a73-4596-b18f-d7fdbbcccca6" (UID: "d754ffb0-1a73-4596-b18f-d7fdbbcccca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:37.578736 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.578695    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr" pod "d754ffb0-1a73-4596-b18f-d7fdbbcccca6" (UID: "d754ffb0-1a73-4596-b18f-d7fdbbcccca6"). InnerVolumeSpecName "kube-api-access-w4xrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:37.675925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.675889    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4xrr\" (UniqueName: \"kubernetes.io/projected/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-kube-api-access-w4xrr\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:37.675925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.675923    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:37.675925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:37.675934    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d754ffb0-1a73-4596-b18f-d7fdbbcccca6-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:38.299411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:38.295726    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn" event={"ID":"d754ffb0-1a73-4596-b18f-d7fdbbcccca6","Type":"ContainerDied","Data":"5f096bca8d2eced9b9c60605438a7899a0655ed07c9281d07aa35cb8a8883785"}
Nov 03 16:21:38.299411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:38.295790    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f096bca8d2eced9b9c60605438a7899a0655ed07c9281d07aa35cb8a8883785"
Nov 03 16:21:38.299411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:38.295859    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"
Nov 03 16:21:38.361411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:38.358677    4964 generic.go:356] "Generic (PLEG): container finished" podID="fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" containerID="15a02e25a714a67de4550afbfecc4d5727952db880596186d6115dc8f1fe2c19" exitCode=0
Nov 03 16:21:38.361411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:38.360075    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29" event={"ID":"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4","Type":"ContainerDied","Data":"15a02e25a714a67de4550afbfecc4d5727952db880596186d6115dc8f1fe2c19"}
Nov 03 16:21:38.680762 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:38.678824    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89826e1-65c6-4480-9eab-e88f239d5ec9" path="/var/lib/kubelet/pods/c89826e1-65c6-4480-9eab-e88f239d5ec9/volumes"
Nov 03 16:21:39.149251 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.149110    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"]
Nov 03 16:21:39.151249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.151222    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d754ffb0-1a73-4596-b18f-d7fdbbcccca6" containerName="revoke-keys"
Nov 03 16:21:39.151433 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.151416    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d754ffb0-1a73-4596-b18f-d7fdbbcccca6" containerName="revoke-keys"
Nov 03 16:21:39.151771 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.151749    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d754ffb0-1a73-4596-b18f-d7fdbbcccca6" containerName="revoke-keys"
Nov 03 16:21:39.156227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.156207    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.170893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.170863    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"]
Nov 03 16:21:39.219801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.219758    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220015 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.219955    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220015 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220004    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220029    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220178    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220261 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220229    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220347 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220327    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220434 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220416    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.220497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.220468    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzns\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.267278 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.267246    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"]
Nov 03 16:21:39.272860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.272834    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.321720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.321685    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.321720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.321726    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.321751    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.321886    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzns\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.321968    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322006    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322336    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322469    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322506    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322582    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322591    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322608    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322668    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322700    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqc5g\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322749    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322782    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322801    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.322901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.322811    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.323707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.323096    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.323707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.323132    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.323707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.323278    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.323707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.323341    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.323707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.323631    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.324978 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.324934    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.326446 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.326428    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.342220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.342189    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.343855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.343824    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzns\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.383003 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.382969    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"]
Nov 03 16:21:39.425009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.424969    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425038    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425073    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqc5g\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425127    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425272    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425307    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425335    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425631 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425461    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.425631 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.425545    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.426571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.426542    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.427849 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.427360    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.428015 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.427968    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.428250 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.428223    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.428336 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.428266    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.428586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.428559    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.428663 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.428624    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.447649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.446832    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.451029 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.450993    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqc5g\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.474351 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.474318    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:39.588786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.588142    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:39.598886 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.598864    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:39.667943 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.667892    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"]
Nov 03 16:21:39.710192 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.710146    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"]
Nov 03 16:21:39.737029 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.736993    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca\") pod \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") "
Nov 03 16:21:39.737453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.737424    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca\") pod \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") "
Nov 03 16:21:39.737703 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.737683    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l\" (UniqueName: \"kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l\") pod \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\" (UID: \"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4\") "
Nov 03 16:21:39.739133 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.739101    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca" pod "fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" (UID: "fb2f81d1-ac41-440e-96f1-bcc075ea7ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:39.740074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.740040    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca" pod "fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" (UID: "fb2f81d1-ac41-440e-96f1-bcc075ea7ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:39.741653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.741625    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l" pod "fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" (UID: "fb2f81d1-ac41-440e-96f1-bcc075ea7ae4"). InnerVolumeSpecName "kube-api-access-fr86l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:39.827643 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.827604    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"]
Nov 03 16:21:39.830922 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.830887    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:39.831070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.830972    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:39.831070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.831014    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:39.840141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.840113    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr86l\" (UniqueName: \"kubernetes.io/projected/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-kube-api-access-fr86l\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.840141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.840140    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.840309 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.840151    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:39.950813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.950716    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"]
Nov 03 16:21:39.962442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.962406    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:39.962529 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.962489    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:39.962587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:39.962530    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:21:40.061757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.061723    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"]
Nov 03 16:21:40.062474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.062453    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" containerName="revoke-keys"
Nov 03 16:21:40.062474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.062473    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" containerName="revoke-keys"
Nov 03 16:21:40.062683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.062670    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" containerName="revoke-keys"
Nov 03 16:21:40.066610 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.066586    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.088710 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.088673    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"]
Nov 03 16:21:40.160600 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.160563    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.160960 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.160943    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkb6l\" (UniqueName: \"kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.163770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.161139    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.277470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.266499    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.278104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.278074    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkb6l\" (UniqueName: \"kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.278365 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.278348    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.287442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.279310    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.296263 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.296231    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.350478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.350165    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkb6l\" (UniqueName: \"kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l\") pod \"maas-api-revoke-sub-eb6f5f43-l8f5s\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.401488 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.401451    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29" event={"ID":"fb2f81d1-ac41-440e-96f1-bcc075ea7ae4","Type":"ContainerDied","Data":"00b233bbec7f20b6c749f869e30e6726eea8b424da80aa4e1c0fc0c97e0464a0"}
Nov 03 16:21:40.401662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.401528    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00b233bbec7f20b6c749f869e30e6726eea8b424da80aa4e1c0fc0c97e0464a0"
Nov 03 16:21:40.401662 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.401595    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"
Nov 03 16:21:40.422420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.420994    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:40.454335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.454295    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" event={"ID":"12ec23d6-4838-498b-bbfc-c95776238998","Type":"ContainerStarted","Data":"ed27238aeedbf09944179c635daef09396382b1a8a717100fafd7d1343dc37bb"}
Nov 03 16:21:40.454602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.454580    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" event={"ID":"12ec23d6-4838-498b-bbfc-c95776238998","Type":"ContainerStarted","Data":"c3a4d284516e349573f76c50ad93dcadecf1cc89ba430428bf79380261022599"}
Nov 03 16:21:40.485113 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.482276    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:40.488885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.488035    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" event={"ID":"89b68376-e690-403a-a38b-52897ffaf73a","Type":"ContainerStarted","Data":"1881b9d20f65cd33e2ab25024197afb2cedc5b77e11dfe777daeee311bbfb5fd"}
Nov 03 16:21:40.488885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.488096    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" event={"ID":"89b68376-e690-403a-a38b-52897ffaf73a","Type":"ContainerStarted","Data":"f2ce79f2c83faa2c6487e19af2e88c60668e62b778b3b0a2e70480946c1b019a"}
Nov 03 16:21:40.517534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.517499    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:40.517832 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.517803    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:40.595107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.586914    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podStartSLOduration=1.586893169 podStartE2EDuration="1.586893169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.564850845 +0000 UTC m=+1162.078232621" watchObservedRunningTime="2026-11-03 16:21:40.586893169 +0000 UTC m=+1162.100274950"
Nov 03 16:21:40.595107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.589920    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:40.595107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:40.591955    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podStartSLOduration=1.5919440360000001 podStartE2EDuration="1.591944036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.515148074 +0000 UTC m=+1162.028529854" watchObservedRunningTime="2026-11-03 16:21:40.591944036 +0000 UTC m=+1162.105325814"
Nov 03 16:21:41.346698 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.346209    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"]
Nov 03 16:21:41.676689 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.675611    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:41.676689 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.675760    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:41.712423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.709717    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:41.712423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.709779    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:41.738423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.728541    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"]
Nov 03 16:21:41.990489 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:41.979594    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.108933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.108897    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"]
Nov 03 16:21:42.160624 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.160568    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s" event={"ID":"b124965c-f0c9-4e4d-a73e-d8f704addbb1","Type":"ContainerStarted","Data":"3c57e551053e8268cc8fb9e469c7646feb21e57bbcd94758d1d3bb785306d3f1"}
Nov 03 16:21:42.178012 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.162559    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:42.196620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.164108    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:42.196620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.166577    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.196620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.166631    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:42.196620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.177902    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:42.208128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.207949    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.283824 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.283578    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" containerID="cri-o://adc8768542b57d3824e49cccd1de30438e153f8389031ed2f06e91873fc68a5d" gracePeriod=45
Nov 03 16:21:42.370507 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.351768    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:42.504565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.504522    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" containerID="cri-o://4d02e42ded70611f545a6b57a540ee20bd58ec809e296cf597d4f87422e150bd" gracePeriod=45
Nov 03 16:21:42.506739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.506709    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:42.506933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.506912    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:42.587412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.542234    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.588041 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.577594    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:42.591911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.589586    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:42.591911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.589632    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:42.591911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.590199    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" containerID="cri-o://b1ce72d683aa584ddd5a05f1ec40044acaf9d2db27a8e0e0d11e1a1efe46fb3f" gracePeriod=45
Nov 03 16:21:42.591911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.590407    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Nov 03 16:21:42.591911 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.590514    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:42.614411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:42.613204    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.094266 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.094217    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.196467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.174629    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 16:21:43.219371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.197059    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.219371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.197978    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.219371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.198633    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.219371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.199209    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.577191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.577147    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.601893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.601858    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:43.602225 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.602194    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:43.687772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.687712    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:43.743535 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.721629    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.765876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.765811    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.788122    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:43.789004    4964 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.175s"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789094    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789229    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789338    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789412    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789473    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789527    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:43.809989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.789578    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:43.836592 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:43.833744    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:44.064706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.025589    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:44.064706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.034134    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:44.064706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.034200    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:44.064706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.036214    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:44.064706 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.036313    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:44.090524 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.065329    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 16:21:44.521412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.497420    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 03 16:21:44.524343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.504230    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:44.525004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.524970    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:44.525171 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.504545    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:44.525559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.504582    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:44.525559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.504495    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:44.595689 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.595633    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.721573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.721511    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.767410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.749063    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Nov 03 16:21:44.767410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.749188    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused"
Nov 03 16:21:44.767410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.749261    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.785125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.784853    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:44.785456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.785422    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:44.825455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.825417    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:44.865845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.865799    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 16:21:44.955411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:44.904669    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" containerID="cri-o://1ca15c243b6147d4794be8440b27daf65be2a2857c4113a88f1b8fc0fd082e05" gracePeriod=42
Nov 03 16:21:45.117421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.113911    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:45.130474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.130435    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:45.497278 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.475726    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:45.498316 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.498265    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:45.659043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.605562    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:45.660123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.607132    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.710442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.608556    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:45.710442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.679790    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:45.710442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.622478    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.710442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.622438    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:45.710442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.622819    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused"
Nov 03 16:21:45.710442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.623513    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Nov 03 16:21:45.795530 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.787540    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 03 16:21:45.895597 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.893532    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:45.895597 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.895184    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 03 16:21:46.072279 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:45.963885    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 16:21:46.213368 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.213326    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:46.281906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.281518    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:46.332730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.321401    4964 generic.go:356] "Generic (PLEG): container finished" podID="300c6366-5c06-4010-a4de-313fe2680332" containerID="b1ce72d683aa584ddd5a05f1ec40044acaf9d2db27a8e0e0d11e1a1efe46fb3f" exitCode=0
Nov 03 16:21:46.332730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.332565    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" event={"ID":"300c6366-5c06-4010-a4de-313fe2680332","Type":"ContainerDied","Data":"b1ce72d683aa584ddd5a05f1ec40044acaf9d2db27a8e0e0d11e1a1efe46fb3f"}
Nov 03 16:21:46.359371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.359164    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" containerName="payload-pre-processing" containerID="cri-o://32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933" gracePeriod=25
Nov 03 16:21:46.370792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.367091    4964 generic.go:356] "Generic (PLEG): container finished" podID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerID="d5d08eba354561e5ecf2404158cedafbf3daeed4b919b1cbe093f8dd2c9f49a5" exitCode=0
Nov 03 16:21:46.370792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.367378    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" event={"ID":"44e31b7d-fda1-45b9-84b2-4295406c3357","Type":"ContainerDied","Data":"d5d08eba354561e5ecf2404158cedafbf3daeed4b919b1cbe093f8dd2c9f49a5"}
Nov 03 16:21:46.370792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.369556    4964 generic.go:356] "Generic (PLEG): container finished" podID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerID="4d02e42ded70611f545a6b57a540ee20bd58ec809e296cf597d4f87422e150bd" exitCode=0
Nov 03 16:21:46.370792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.369668    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" event={"ID":"5c4e6063-5b41-4e2d-a256-e009c9d5adcf","Type":"ContainerDied","Data":"4d02e42ded70611f545a6b57a540ee20bd58ec809e296cf597d4f87422e150bd"}
Nov 03 16:21:46.371610 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.371585    4964 generic.go:356] "Generic (PLEG): container finished" podID="b124965c-f0c9-4e4d-a73e-d8f704addbb1" containerID="86f5ca7564789af6f82fe2db88c736e4535f13fdf6379c0159cbc8302a62e2a8" exitCode=0
Nov 03 16:21:46.371820 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.371803    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s" event={"ID":"b124965c-f0c9-4e4d-a73e-d8f704addbb1","Type":"ContainerDied","Data":"86f5ca7564789af6f82fe2db88c736e4535f13fdf6379c0159cbc8302a62e2a8"}
Nov 03 16:21:46.375371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.375346    4964 generic.go:356] "Generic (PLEG): container finished" podID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerID="adc8768542b57d3824e49cccd1de30438e153f8389031ed2f06e91873fc68a5d" exitCode=0
Nov 03 16:21:46.376379 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.376356    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" event={"ID":"59bf2433-3ac6-4208-b49c-0cfea8472d5e","Type":"ContainerDied","Data":"adc8768542b57d3824e49cccd1de30438e153f8389031ed2f06e91873fc68a5d"}
Nov 03 16:21:46.440422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.439367    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.524410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.520956    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"]
Nov 03 16:21:46.524410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.521200    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" containerID="cri-o://9403a1f5f6316265e3c78d8ec04c788a04839f6bef2c82bd7b0766031a1b8a84" gracePeriod=30
Nov 03 16:21:46.541751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.541720    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:46.541960 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.541931    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:46.556665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.556626    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Nov 03 16:21:46.591098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.591006    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:46.591345 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.591316    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:46.604050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.604009    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"]
Nov 03 16:21:46.604803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.604771    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" containerID="cri-o://0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d" gracePeriod=30
Nov 03 16:21:46.607231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.607156    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused"
Nov 03 16:21:46.671432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.663701    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.671432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.665127    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:46.675446 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.674195    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s" podStartSLOduration=6.674174011 podStartE2EDuration="6.674174011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:45.457261299 +0000 UTC m=+1166.970643089" watchObservedRunningTime="2026-11-03 16:21:46.674174011 +0000 UTC m=+1168.187555788"
Nov 03 16:21:46.709618 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.688531    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:46.729358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.729308    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerName="payload-processing" containerID="cri-o://abfbe3daaf93bd704c692b1768836f2ee684de5f103c8318e84d048e0d90dd92" gracePeriod=25
Nov 03 16:21:46.735418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.731542    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"]
Nov 03 16:21:46.748077 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.748033    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy" containerID="cri-o://34812bf92e76f534571ada6c536e7f506276400aacacff72cea2519a09ae389a" gracePeriod=30
Nov 03 16:21:46.885735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.884909    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 16:21:46.907108 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:46.906503    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.163351 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.163302    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:47.187614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.187572    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:47.495473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.476815    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:47.553413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.503055    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:47.554233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.553749    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:47.732724 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.731063    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:47.995838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:47.995718    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"]
Nov 03 16:21:48.061428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.052145    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused"
Nov 03 16:21:48.061428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.053957    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.061428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.055297    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:48.061428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.055349    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:48.061428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.056050    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.075874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.075834    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:48.085633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.082376    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.085633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.077654    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 16:21:48.085633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.076293    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.163478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.163435    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:48.167521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.167467    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.171500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.169333    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:48.205412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.176792    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.237438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.210586    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:48.237438 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.212709    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:48.237758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.237557    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:48.237758 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.237620    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.424661 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.424622    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" containerID="cri-o://a32790e4514bdb37a2c593d4004f3a215edb5525978e0b62c9ce7eb3fd5849fe" gracePeriod=45
Nov 03 16:21:48.468844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.426552    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:48.550242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.451700    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir\") pod \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") "
Nov 03 16:21:48.551230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.550653    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home\") pod \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") "
Nov 03 16:21:48.551454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.551436    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache\") pod \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") "
Nov 03 16:21:48.551859 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.551842    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm\") pod \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") "
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.661586    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs\") pod \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") "
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.661731    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j\" (UniqueName: \"kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j\") pod \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\" (UID: \"59bf2433-3ac6-4208-b49c-0cfea8472d5e\") "
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.549091    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.451987    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir" pod "59bf2433-3ac6-4208-b49c-0cfea8472d5e" (UID: "59bf2433-3ac6-4208-b49c-0cfea8472d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.551178    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home" pod "59bf2433-3ac6-4208-b49c-0cfea8472d5e" (UID: "59bf2433-3ac6-4208-b49c-0cfea8472d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.575296    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:49.665615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.663269    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:49.688596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.681800    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:49.688596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.681885    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.688596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.506274    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.693443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.505836    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:49.693683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.638484    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.711155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.620587    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 16:21:49.711953 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.695140    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:49.715212 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.712205    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:49.776465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.478282    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 16:21:49.776465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.014666    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" containerID="cri-o://40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef" gracePeriod=44
Nov 03 16:21:49.776465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.724528    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache" pod "59bf2433-3ac6-4208-b49c-0cfea8472d5e" (UID: "59bf2433-3ac6-4208-b49c-0cfea8472d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.642005    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.878364    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.296062    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.340144    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.782421    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.889332    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.889419    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:48.773821    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.890222    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.296167    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.890317    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.899538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.362666    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:49.983413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.900624    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j" pod "59bf2433-3ac6-4208-b49c-0cfea8472d5e" (UID: "59bf2433-3ac6-4208-b49c-0cfea8472d5e"). InnerVolumeSpecName "kube-api-access-c5l4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:50.009568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.988083    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.009568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.988127    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.009568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.988145    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.009568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:49.988170    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5l4j\" (UniqueName: \"kubernetes.io/projected/59bf2433-3ac6-4208-b49c-0cfea8472d5e-kube-api-access-c5l4j\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.121694 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.093195    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm" pod "59bf2433-3ac6-4208-b49c-0cfea8472d5e" (UID: "59bf2433-3ac6-4208-b49c-0cfea8472d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:50.129974 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.122409    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" containerID="cri-o://d514fc8d7a0d69b6be01137bfeb591437c97582ac7bcb40f8492f2ad3fc0243d" gracePeriod=45
Nov 03 16:21:50.144459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.143064    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs" pod "59bf2433-3ac6-4208-b49c-0cfea8472d5e" (UID: "59bf2433-3ac6-4208-b49c-0cfea8472d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:50.402809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.402763    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:50.476881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.405031    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf2433-3ac6-4208-b49c-0cfea8472d5e-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.476881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.476191    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf2433-3ac6-4208-b49c-0cfea8472d5e-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:50.533421 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:50.526752    4964 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.889s"
Nov 03 16:21:50.533421 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.526968    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:50.642412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.637409    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"
Nov 03 16:21:50.675459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.670623    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:50.763943 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.756661    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:50.781503 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.777463    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:50.815485 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.727115    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:50.815766 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.815737    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:50.820193 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.807236    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:50.860939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.842837    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused"
Nov 03 16:21:50.862527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.862502    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:50.865685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.862734    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw" event={"ID":"59bf2433-3ac6-4208-b49c-0cfea8472d5e","Type":"ContainerDied","Data":"a6c9851963dfa408968821f0214bc577a9d753f2221f10500ec6541c715cc011"}
Nov 03 16:21:50.866226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.866205    4964 scope.go:122] "RemoveContainer" containerID="adc8768542b57d3824e49cccd1de30438e153f8389031ed2f06e91873fc68a5d"
Nov 03 16:21:50.926596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.926484    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:50.948819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.948776    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 16:21:50.969743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.969707    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:51.033409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.002999    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:51.033409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.003196    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:51.033409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:50.958709    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:51.033409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.005841    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused"
Nov 03 16:21:51.050453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.039856    4964 generic.go:356] "Generic (PLEG): container finished" podID="caec860b-340b-4777-9b61-b525bb9c42f6" containerID="1ca15c243b6147d4794be8440b27daf65be2a2857c4113a88f1b8fc0fd082e05" exitCode=0
Nov 03 16:21:51.051151 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.051121    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:51.100460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.053505    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:51.172669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.100727    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:21:51.172669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.100785    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:21:51.172669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.101277    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:51.240038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.239991    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused"
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.260472    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache\") pod \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") "
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.260546    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm\") pod \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") "
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.260650    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home\") pod \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") "
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.260743    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn\" (UniqueName: \"kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn\") pod \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") "
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.260892    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs\") pod \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") "
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.260936    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir\") pod \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\" (UID: \"5c4e6063-5b41-4e2d-a256-e009c9d5adcf\") "
Nov 03 16:21:51.292496 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.273911    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache" pod "5c4e6063-5b41-4e2d-a256-e009c9d5adcf" (UID: "5c4e6063-5b41-4e2d-a256-e009c9d5adcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:51.320233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.293116    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:51.320233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.293682    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir" pod "5c4e6063-5b41-4e2d-a256-e009c9d5adcf" (UID: "5c4e6063-5b41-4e2d-a256-e009c9d5adcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:51.339581 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.337620    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused"
Nov 03 16:21:51.450750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.450669    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home" pod "5c4e6063-5b41-4e2d-a256-e009c9d5adcf" (UID: "5c4e6063-5b41-4e2d-a256-e009c9d5adcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:51.451837 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.451788    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": EOF"
Nov 03 16:21:51.634533 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.634486    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:51.638296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.638265    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:51.638532 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.638501    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:51.653756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.653723    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:51.653994 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.653970    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:51.679990 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.679946    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:51.797414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.789733    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn" pod "5c4e6063-5b41-4e2d-a256-e009c9d5adcf" (UID: "5c4e6063-5b41-4e2d-a256-e009c9d5adcf"). InnerVolumeSpecName "kube-api-access-spzfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:51.835926 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.829026    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm" pod "5c4e6063-5b41-4e2d-a256-e009c9d5adcf" (UID: "5c4e6063-5b41-4e2d-a256-e009c9d5adcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:51.930413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.923653    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:51.987428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:51.986727    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:52.010432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.010375    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": dial tcp 10.132.0.59:8000: connect: connection refused"
Nov 03 16:21:52.070648 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:52.035096    4964 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.169s"
Nov 03 16:21:52.070648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.035209    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" event={"ID":"caec860b-340b-4777-9b61-b525bb9c42f6","Type":"ContainerDied","Data":"1ca15c243b6147d4794be8440b27daf65be2a2857c4113a88f1b8fc0fd082e05"}
Nov 03 16:21:52.070648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.035279    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:21:52.070648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.035315    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:52.072482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.071321    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs" pod "5c4e6063-5b41-4e2d-a256-e009c9d5adcf" (UID: "5c4e6063-5b41-4e2d-a256-e009c9d5adcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:52.148743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.082296    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:52.148743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.091757    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:52.148743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.144982    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:21:52.148743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.148527    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:52.157728 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.157693    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:21:52.161738 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.161668    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused"
Nov 03 16:21:52.178415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.171296    4964 generic.go:356] "Generic (PLEG): container finished" podID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerID="a32790e4514bdb37a2c593d4004f3a215edb5525978e0b62c9ce7eb3fd5849fe" exitCode=0
Nov 03 16:21:52.178415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.171379    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" event={"ID":"98c8edbf-14c3-4af7-8f15-e0b4b919b625","Type":"ContainerDied","Data":"a32790e4514bdb37a2c593d4004f3a215edb5525978e0b62c9ce7eb3fd5849fe"}
Nov 03 16:21:52.181406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.181132    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf" event={"ID":"300c6366-5c06-4010-a4de-313fe2680332","Type":"ContainerDied","Data":"802cb54dbf28a5ca2dbf5318b22b114c2b49ab875eafb70ea7116d6e3fb2ccd4"}
Nov 03 16:21:52.181406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.181236    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"
Nov 03 16:21:52.187958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.187908    4964 scope.go:122] "RemoveContainer" containerID="b1ce72d683aa584ddd5a05f1ec40044acaf9d2db27a8e0e0d11e1a1efe46fb3f"
Nov 03 16:21:52.188777 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.188626    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Nov 03 16:21:52.192728 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.192694    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"]
Nov 03 16:21:52.197557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.196896    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5" event={"ID":"44e31b7d-fda1-45b9-84b2-4295406c3357","Type":"ContainerDied","Data":"92ba80ca71de92f32fb1e7b46951b6a696af2f36a5b2c573a89ed463dcc2db86"}
Nov 03 16:21:52.197557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.197006    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"
Nov 03 16:21:52.200462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.200437    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.200575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.200465    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.200575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.200481    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.200575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.200495    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.200575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.200508    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.200575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.200521    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spzfn\" (UniqueName: \"kubernetes.io/projected/5c4e6063-5b41-4e2d-a256-e009c9d5adcf-kube-api-access-spzfn\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.201497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.201473    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" event={"ID":"e8a6b2b1-84b1-43df-8495-85c3d904a0d7","Type":"ContainerDied","Data":"d514fc8d7a0d69b6be01137bfeb591437c97582ac7bcb40f8492f2ad3fc0243d"}
Nov 03 16:21:52.226907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.218544    4964 generic.go:356] "Generic (PLEG): container finished" podID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerID="d514fc8d7a0d69b6be01137bfeb591437c97582ac7bcb40f8492f2ad3fc0243d" exitCode=0
Nov 03 16:21:52.263493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.263452    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-2-w3-d7196a-kserve-7666f54fd7-hnghw"]
Nov 03 16:21:52.334489 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.307987    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache\") pod \"300c6366-5c06-4010-a4de-313fe2680332\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.334907    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home\") pod \"300c6366-5c06-4010-a4de-313fe2680332\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.334975    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca\") pod \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335040    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l\" (UniqueName: \"kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l\") pod \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335092    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm\") pod \"300c6366-5c06-4010-a4de-313fe2680332\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335188    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir\") pod \"300c6366-5c06-4010-a4de-313fe2680332\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335221    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f\" (UniqueName: \"kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f\") pod \"44e31b7d-fda1-45b9-84b2-4295406c3357\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335256    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7\" (UniqueName: \"kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7\") pod \"300c6366-5c06-4010-a4de-313fe2680332\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335289    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca\") pod \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\" (UID: \"b124965c-f0c9-4e4d-a73e-d8f704addbb1\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335381    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs\" (UniqueName: \"kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs\") pod \"300c6366-5c06-4010-a4de-313fe2680332\" (UID: \"300c6366-5c06-4010-a4de-313fe2680332\") "
Nov 03 16:21:52.349377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.335480    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume\") pod \"44e31b7d-fda1-45b9-84b2-4295406c3357\" (UID: \"44e31b7d-fda1-45b9-84b2-4295406c3357\") "
Nov 03 16:21:52.383441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.350415    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume" pod "44e31b7d-fda1-45b9-84b2-4295406c3357" (UID: "44e31b7d-fda1-45b9-84b2-4295406c3357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:52.383441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.359511    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache" pod "300c6366-5c06-4010-a4de-313fe2680332" (UID: "300c6366-5c06-4010-a4de-313fe2680332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:52.383441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.373524    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home" pod "300c6366-5c06-4010-a4de-313fe2680332" (UID: "300c6366-5c06-4010-a4de-313fe2680332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.462370    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"]
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463433    4964 cpu_manager.go:414] "containerMap: removing container" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerName="payload-processing"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463456    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerName="payload-processing"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463472    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b124965c-f0c9-4e4d-a73e-d8f704addbb1" containerName="revoke-keys"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463479    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b124965c-f0c9-4e4d-a73e-d8f704addbb1" containerName="revoke-keys"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463494    4964 cpu_manager.go:414] "containerMap: removing container" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463502    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463545    4964 cpu_manager.go:414] "containerMap: removing container" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463561    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463605    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463612    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463788    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="300c6366-5c06-4010-a4de-313fe2680332" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463805    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" containerName="main"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463816    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" containerName="payload-processing"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463828    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b124965c-f0c9-4e4d-a73e-d8f704addbb1" containerName="revoke-keys"
Nov 03 16:21:52.480657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.463838    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" containerName="main"
Nov 03 16:21:52.483459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.483427    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:52.493054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.492544    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:52.493054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.491135    4964 scope.go:122] "RemoveContainer" containerID="d5d08eba354561e5ecf2404158cedafbf3daeed4b919b1cbe093f8dd2c9f49a5"
Nov 03 16:21:52.507545 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.491509    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir" pod "300c6366-5c06-4010-a4de-313fe2680332" (UID: "300c6366-5c06-4010-a4de-313fe2680332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:52.511541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.510251    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.511541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.510284    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e31b7d-fda1-45b9-84b2-4295406c3357-plugins-config-volume\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.511541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.510311    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.511541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.510328    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.537071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.535895    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused"
Nov 03 16:21:52.552534 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.537914    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"
Nov 03 16:21:52.588993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.588623    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:52.588993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.588684    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:52.589937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.589860    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6" event={"ID":"5c4e6063-5b41-4e2d-a256-e009c9d5adcf","Type":"ContainerDied","Data":"71661ee4bfc8f2c39d4bbf44a5cab6e697da3b79ce5ad125da8afd969b92b5d6"}
Nov 03 16:21:52.591311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.590190    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.612139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.606488    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca" pod "b124965c-f0c9-4e4d-a73e-d8f704addbb1" (UID: "b124965c-f0c9-4e4d-a73e-d8f704addbb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:52.612335 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.612138    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.625817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.625787    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"
Nov 03 16:21:52.647707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.647618    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca" pod "b124965c-f0c9-4e4d-a73e-d8f704addbb1" (UID: "b124965c-f0c9-4e4d-a73e-d8f704addbb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:52.720445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.713276    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.720445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.713374    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qm8s\" (UniqueName: \"kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.720445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.713743    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.720445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.713854    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b124965c-f0c9-4e4d-a73e-d8f704addbb1-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.897772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.814956    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.897772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.815066    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.897772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.815122    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qm8s\" (UniqueName: \"kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.897772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.855568    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.897772 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:52.858562    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a6b2b1_84b1_43df_8495_85c3d904a0d7.slice/crio-conmon-d514fc8d7a0d69b6be01137bfeb591437c97582ac7bcb40f8492f2ad3fc0243d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89127b8_55cb_4686_9d16_7009d7cca6be.slice/crio-40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:21:52.984572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.915655    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:52.984572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.923012    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qm8s\" (UniqueName: \"kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:52.984572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.923639    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs" pod "300c6366-5c06-4010-a4de-313fe2680332" (UID: "300c6366-5c06-4010-a4de-313fe2680332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:52.984572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.943523    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:52.984572 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.975571    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7" pod "300c6366-5c06-4010-a4de-313fe2680332" (UID: "300c6366-5c06-4010-a4de-313fe2680332"). InnerVolumeSpecName "kube-api-access-tmxk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:52.994827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.989259    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxk7\" (UniqueName: \"kubernetes.io/projected/300c6366-5c06-4010-a4de-313fe2680332-kube-api-access-tmxk7\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:52.994827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:52.989294    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300c6366-5c06-4010-a4de-313fe2680332-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.154296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.003144    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerName="payload-pre-processing" containerID="cri-o://703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6" gracePeriod=25
Nov 03 16:21:53.154296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.005555    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f" pod "44e31b7d-fda1-45b9-84b2-4295406c3357" (UID: "44e31b7d-fda1-45b9-84b2-4295406c3357"). InnerVolumeSpecName "kube-api-access-x4f8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.154296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.005648    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm" pod "300c6366-5c06-4010-a4de-313fe2680332" (UID: "300c6366-5c06-4010-a4de-313fe2680332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.154296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.005759    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l" pod "b124965c-f0c9-4e4d-a73e-d8f704addbb1" (UID: "b124965c-f0c9-4e4d-a73e-d8f704addbb1"). InnerVolumeSpecName "kube-api-access-jkb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:53.154296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.029642    4964 scope.go:122] "RemoveContainer" containerID="4d02e42ded70611f545a6b57a540ee20bd58ec809e296cf597d4f87422e150bd"
Nov 03 16:21:53.218296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.195055    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:53.218296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.204368    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkb6l\" (UniqueName: \"kubernetes.io/projected/b124965c-f0c9-4e4d-a73e-d8f704addbb1-kube-api-access-jkb6l\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.218296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.204426    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300c6366-5c06-4010-a4de-313fe2680332-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.218296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.204457    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4f8f\" (UniqueName: \"kubernetes.io/projected/44e31b7d-fda1-45b9-84b2-4295406c3357-kube-api-access-x4f8f\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.351741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.337242    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bf2433-3ac6-4208-b49c-0cfea8472d5e" path="/var/lib/kubelet/pods/59bf2433-3ac6-4208-b49c-0cfea8472d5e/volumes"
Nov 03 16:21:53.351741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.339056    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s" event={"ID":"b124965c-f0c9-4e4d-a73e-d8f704addbb1","Type":"ContainerDied","Data":"3c57e551053e8268cc8fb9e469c7646feb21e57bbcd94758d1d3bb785306d3f1"}
Nov 03 16:21:53.351741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.339126    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c57e551053e8268cc8fb9e469c7646feb21e57bbcd94758d1d3bb785306d3f1"
Nov 03 16:21:53.351741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.339155    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"]
Nov 03 16:21:53.351741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.339216    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"]
Nov 03 16:21:53.511475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.499489    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:53.576509 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.574779    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"]
Nov 03 16:21:53.576509 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.574937    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:53.659466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.632548    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused"
Nov 03 16:21:53.659466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.633441    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:53.659466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.633490    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:53.659466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.633515    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afeece82-qcfp5\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:53.686594 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.678661    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:53.686594 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.678733    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:53.691671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.691611    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:53.743004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.741461    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.807864    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir\") pod \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.807933    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home\") pod \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.807995    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x\" (UniqueName: \"kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x\") pod \"caec860b-340b-4777-9b61-b525bb9c42f6\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.808030    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home\") pod \"caec860b-340b-4777-9b61-b525bb9c42f6\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.808082    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache\") pod \"caec860b-340b-4777-9b61-b525bb9c42f6\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.808202    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm\") pod \"caec860b-340b-4777-9b61-b525bb9c42f6\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.808268    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm\") pod \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") "
Nov 03 16:21:53.809352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.808343    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26\" (UniqueName: \"kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26\") pod \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") "
Nov 03 16:21:53.820654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.820619    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:53.847172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.847128    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home" pod "98c8edbf-14c3-4af7-8f15-e0b4b919b625" (UID: "98c8edbf-14c3-4af7-8f15-e0b4b919b625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.847860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.847817    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home" pod "caec860b-340b-4777-9b61-b525bb9c42f6" (UID: "caec860b-340b-4777-9b61-b525bb9c42f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.860942 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.859605    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir" pod "98c8edbf-14c3-4af7-8f15-e0b4b919b625" (UID: "98c8edbf-14c3-4af7-8f15-e0b4b919b625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.871453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.864955    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache\") pod \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") "
Nov 03 16:21:53.871453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.865070    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir\") pod \"caec860b-340b-4777-9b61-b525bb9c42f6\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") "
Nov 03 16:21:53.871453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.865124    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs\") pod \"caec860b-340b-4777-9b61-b525bb9c42f6\" (UID: \"caec860b-340b-4777-9b61-b525bb9c42f6\") "
Nov 03 16:21:53.871453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.865218    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs\" (UniqueName: \"kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs\") pod \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\" (UID: \"98c8edbf-14c3-4af7-8f15-e0b4b919b625\") "
Nov 03 16:21:53.890036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.873045    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:53.892220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.892188    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9xv\" (UniqueName: \"kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.960438    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.960704    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.960725    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.960738    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.958586    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.892593    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir" pod "caec860b-340b-4777-9b61-b525bb9c42f6" (UID: "caec860b-340b-4777-9b61-b525bb9c42f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:53.984845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:53.936962    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache" pod "98c8edbf-14c3-4af7-8f15-e0b4b919b625" (UID: "98c8edbf-14c3-4af7-8f15-e0b4b919b625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.023744 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.023650    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache" pod "caec860b-340b-4777-9b61-b525bb9c42f6" (UID: "caec860b-340b-4777-9b61-b525bb9c42f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.099317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.099271    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 16:21:54.103456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.102766    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:54.108100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.108057    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home\") pod \"b89127b8-55cb-4686-9d16-7009d7cca6be\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") "
Nov 03 16:21:54.108477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.108434    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm\") pod \"b89127b8-55cb-4686-9d16-7009d7cca6be\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") "
Nov 03 16:21:54.108704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.108677    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache\") pod \"b89127b8-55cb-4686-9d16-7009d7cca6be\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") "
Nov 03 16:21:54.108884 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.108868    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir\") pod \"b89127b8-55cb-4686-9d16-7009d7cca6be\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") "
Nov 03 16:21:54.109026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.109011    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs\" (UniqueName: \"kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs\") pod \"b89127b8-55cb-4686-9d16-7009d7cca6be\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") "
Nov 03 16:21:54.109234 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.109210    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644\" (UniqueName: \"kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644\") pod \"b89127b8-55cb-4686-9d16-7009d7cca6be\" (UID: \"b89127b8-55cb-4686-9d16-7009d7cca6be\") "
Nov 03 16:21:54.110014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.109986    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.110832 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.110802    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.111094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.111067    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9xv\" (UniqueName: \"kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.111419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.111378    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.111515 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.111504    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.111617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.111602    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.112358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.112327    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home" pod "b89127b8-55cb-4686-9d16-7009d7cca6be" (UID: "b89127b8-55cb-4686-9d16-7009d7cca6be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.129493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.128475    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache" pod "b89127b8-55cb-4686-9d16-7009d7cca6be" (UID: "b89127b8-55cb-4686-9d16-7009d7cca6be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.129493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.128744    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir" pod "b89127b8-55cb-4686-9d16-7009d7cca6be" (UID: "b89127b8-55cb-4686-9d16-7009d7cca6be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.210408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.206439    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.216869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.216546    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:54.299036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.217828    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2" event={"ID":"caec860b-340b-4777-9b61-b525bb9c42f6","Type":"ContainerDied","Data":"6f3c3d1af6a4bf0a25a74ffd92b9265f9f274097bc6d9b2db8715cdb7f620528"}
Nov 03 16:21:54.299036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.217900    4964 scope.go:122] "RemoveContainer" containerID="1ca15c243b6147d4794be8440b27daf65be2a2857c4113a88f1b8fc0fd082e05"
Nov 03 16:21:54.299036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.218075    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"
Nov 03 16:21:54.299036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.228420    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9xv\" (UniqueName: \"kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.299036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.230106    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"]
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.254208    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.330034    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs" pod "b89127b8-55cb-4686-9d16-7009d7cca6be" (UID: "b89127b8-55cb-4686-9d16-7009d7cca6be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.330136    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm" pod "b89127b8-55cb-4686-9d16-7009d7cca6be" (UID: "b89127b8-55cb-4686-9d16-7009d7cca6be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.330232    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs" pod "98c8edbf-14c3-4af7-8f15-e0b4b919b625" (UID: "98c8edbf-14c3-4af7-8f15-e0b4b919b625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.330325    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644" pod "b89127b8-55cb-4686-9d16-7009d7cca6be" (UID: "b89127b8-55cb-4686-9d16-7009d7cca6be"). InnerVolumeSpecName "kube-api-access-dl644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.330752    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fccc1b-wt5n2\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.353412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.341880    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm" pod "98c8edbf-14c3-4af7-8f15-e0b4b919b625" (UID: "98c8edbf-14c3-4af7-8f15-e0b4b919b625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.388457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.355111    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm" pod "caec860b-340b-4777-9b61-b525bb9c42f6" (UID: "caec860b-340b-4777-9b61-b525bb9c42f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.388457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.380969    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26" pod "98c8edbf-14c3-4af7-8f15-e0b4b919b625" (UID: "98c8edbf-14c3-4af7-8f15-e0b4b919b625"). InnerVolumeSpecName "kube-api-access-ptn26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:54.388755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.388666    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs" pod "caec860b-340b-4777-9b61-b525bb9c42f6" (UID: "caec860b-340b-4777-9b61-b525bb9c42f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:54.433728 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.419066    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x" pod "caec860b-340b-4777-9b61-b525bb9c42f6" (UID: "caec860b-340b-4777-9b61-b525bb9c42f6"). InnerVolumeSpecName "kube-api-access-8l46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:54.434836 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.425577    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused"
Nov 03 16:21:54.438741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.438707    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c8edbf-14c3-4af7-8f15-e0b4b919b625-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.438947 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.438928    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439033    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439130    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l46x\" (UniqueName: \"kubernetes.io/projected/caec860b-340b-4777-9b61-b525bb9c42f6-kube-api-access-8l46x\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439222 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439211    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439293    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89127b8-55cb-4686-9d16-7009d7cca6be-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439398 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439369    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caec860b-340b-4777-9b61-b525bb9c42f6-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439487 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439474    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl644\" (UniqueName: \"kubernetes.io/projected/b89127b8-55cb-4686-9d16-7009d7cca6be-kube-api-access-dl644\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439592 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439580    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c8edbf-14c3-4af7-8f15-e0b4b919b625-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439667    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89127b8-55cb-4686-9d16-7009d7cca6be-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439763 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439752    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptn26\" (UniqueName: \"kubernetes.io/projected/98c8edbf-14c3-4af7-8f15-e0b4b919b625-kube-api-access-ptn26\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.439845 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.439834    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caec860b-340b-4777-9b61-b525bb9c42f6-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:54.452566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.451985    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/simulator-w3-d7196a-kserve-5bc994779b-bjgj6"]
Nov 03 16:21:54.461353 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.460765    4964 generic.go:356] "Generic (PLEG): container finished" podID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerID="9403a1f5f6316265e3c78d8ec04c788a04839f6bef2c82bd7b0766031a1b8a84" exitCode=0
Nov 03 16:21:54.461353 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.460881    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" event={"ID":"1d4d726f-ef35-4781-8c27-431dad4ab68a","Type":"ContainerDied","Data":"9403a1f5f6316265e3c78d8ec04c788a04839f6bef2c82bd7b0766031a1b8a84"}
Nov 03 16:21:54.463886 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.463853    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh" event={"ID":"98c8edbf-14c3-4af7-8f15-e0b4b919b625","Type":"ContainerDied","Data":"8a18fb6ed6a6208e8eb260b9c3c313f3f9aa59ed095416f08ffa010b8bd9d393"}
Nov 03 16:21:54.464095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.464080    4964 scope.go:122] "RemoveContainer" containerID="a32790e4514bdb37a2c593d4004f3a215edb5525978e0b62c9ce7eb3fd5849fe"
Nov 03 16:21:54.464358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.464340    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"
Nov 03 16:21:54.478770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.478742    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:54.480684 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.480625    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:54.482686 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.482656    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:54.482870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.482843    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:54.491179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.490953    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:54.515487 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.514328    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:21:54.582128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.582088    4964 generic.go:356] "Generic (PLEG): container finished" podID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerID="40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef" exitCode=0
Nov 03 16:21:54.582412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.582368    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" event={"ID":"b89127b8-55cb-4686-9d16-7009d7cca6be","Type":"ContainerDied","Data":"40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef"}
Nov 03 16:21:54.628603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.628542    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk" event={"ID":"b89127b8-55cb-4686-9d16-7009d7cca6be","Type":"ContainerDied","Data":"1a215028ea7e9e8f150c41161c8492c2fd89d1473f1c7fd6cec527e2191868a2"}
Nov 03 16:21:54.632407 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.582560    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"
Nov 03 16:21:54.646249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.589047    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:54.646249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.642837    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:54.754209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.754093    4964 scope.go:122] "RemoveContainer" containerID="40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef"
Nov 03 16:21:54.790442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.789807    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" containerID="cri-o://65023ca49df9f0bf7f11daf40c81b27bef29809bc4b7392f50dc050d8e55e705" gracePeriod=45
Nov 03 16:21:54.794210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.794162    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" containerID="cri-o://21488714470cf06a5d0b4c54dec18245c5c871ca982ec5358b82fef7de96c02e" gracePeriod=45
Nov 03 16:21:54.796462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.796077    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.903965    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904047    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904087    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904150    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904213    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904267    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904297    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904448    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.931890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.904567    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo\") pod \"1d4d726f-ef35-4781-8c27-431dad4ab68a\" (UID: \"1d4d726f-ef35-4781-8c27-431dad4ab68a\") "
Nov 03 16:21:54.975796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.975751    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4e6063-5b41-4e2d-a256-e009c9d5adcf" path="/var/lib/kubelet/pods/5c4e6063-5b41-4e2d-a256-e009c9d5adcf/volumes"
Nov 03 16:21:54.989677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.983617    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:54.989677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.985641    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.989677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.987621    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.989677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.987710    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": EOF"
Nov 03 16:21:54.989677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.988636    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.989677 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.988821    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:54.992080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.991531    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": dial tcp 10.132.0.54:8000: connect: connection refused"
Nov 03 16:21:54.996986 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.996846    4964 scope.go:122] "RemoveContainer" containerID="40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef"
Nov 03 16:21:54.997566 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:54.997489    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef\": container with ID starting with 40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef not found: ID does not exist" containerID="40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef"
Nov 03 16:21:54.997566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:54.997526    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef"} err="failed to get container status \"40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef\": rpc error: code = NotFound desc = could not find container \"40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef\": container with ID starting with 40cc1261913a9c7fa228cb55b3b3659491c330a8c3f9281dc67a7d5d42ac64ef not found: ID does not exist"
Nov 03 16:21:55.002039 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.001990    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.072671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.063614    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.072671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.063649    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.072671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.063668    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.072671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.063683    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.072671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.063698    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4d726f-ef35-4781-8c27-431dad4ab68a-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.072671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.063713    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4d726f-ef35-4781-8c27-431dad4ab68a-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.095447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.094723    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"]
Nov 03 16:21:55.095447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.094836    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d569fb-68574c7bc-skvr5"]
Nov 03 16:21:55.095447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.094869    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:55.095447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.094897    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"]
Nov 03 16:21:55.095447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.094916    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/distinct-w3-d7196a-kserve-6d8fd9c448-4h6gf"]
Nov 03 16:21:55.098558 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.098511    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.098909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.098865    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:21:55.101539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.101514    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj" pod "1d4d726f-ef35-4781-8c27-431dad4ab68a" (UID: "1d4d726f-ef35-4781-8c27-431dad4ab68a"). InnerVolumeSpecName "kube-api-access-gh8rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.112587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.112543    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"]
Nov 03 16:21:55.180452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.180034    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8rj\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-kube-api-access-gh8rj\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.180452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.180075    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.180452 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.180091    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4d726f-ef35-4781-8c27-431dad4ab68a-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.181126 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.181101    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7196a/premium-w3-d7196a-kserve-7cc8557f6f-k9wm2"]
Nov 03 16:21:55.196461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.188255    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:55.196461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.188946    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"]
Nov 03 16:21:55.196698 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.196646    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/premium-w1-c09b17-kserve-7664bcd5fc-tm2hh"]
Nov 03 16:21:55.276506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.271589    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"]
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.280797    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr\" (UniqueName: \"kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr\") pod \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") "
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.280865    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir\") pod \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") "
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.280918    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs\") pod \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") "
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.281054    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm\") pod \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") "
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.281086    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home\") pod \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") "
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.281114    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache\") pod \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\" (UID: \"e8a6b2b1-84b1-43df-8495-85c3d904a0d7\") "
Nov 03 16:21:55.286010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.285687    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache" pod "e8a6b2b1-84b1-43df-8495-85c3d904a0d7" (UID: "e8a6b2b1-84b1-43df-8495-85c3d904a0d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.337605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.337216    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir" pod "e8a6b2b1-84b1-43df-8495-85c3d904a0d7" (UID: "e8a6b2b1-84b1-43df-8495-85c3d904a0d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.341550 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.341519    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/simulator-w1-c09b17-kserve-6dc845f788-fk9sk"]
Nov 03 16:21:55.362016 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.361875    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home" pod "e8a6b2b1-84b1-43df-8495-85c3d904a0d7" (UID: "e8a6b2b1-84b1-43df-8495-85c3d904a0d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.377468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.377409    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr" pod "e8a6b2b1-84b1-43df-8495-85c3d904a0d7" (UID: "e8a6b2b1-84b1-43df-8495-85c3d904a0d7"). InnerVolumeSpecName "kube-api-access-626sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.385197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.384491    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626sr\" (UniqueName: \"kubernetes.io/projected/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-kube-api-access-626sr\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.385197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.384532    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.385197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.384547    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.385197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.384563    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.387184 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.387119    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs" pod "e8a6b2b1-84b1-43df-8495-85c3d904a0d7" (UID: "e8a6b2b1-84b1-43df-8495-85c3d904a0d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:55.387756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.387709    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm" pod "e8a6b2b1-84b1-43df-8495-85c3d904a0d7" (UID: "e8a6b2b1-84b1-43df-8495-85c3d904a0d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.477891 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.477728    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:55.477891 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.477789    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:55.488227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.488157    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.488227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.488197    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a6b2b1-84b1-43df-8495-85c3d904a0d7-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.493292 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.492518    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:21:55.654855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.654475    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.654855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.654540    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.654855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.654573    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.654855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.654602    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.654855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.654662    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.656801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.655372    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.656801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.655447    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.656801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.655588    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.656801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.655786    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs\") pod \"077b8c38-65b7-443c-afe5-8d43d20f0a40\" (UID: \"077b8c38-65b7-443c-afe5-8d43d20f0a40\") "
Nov 03 16:21:55.657232 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.657196    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.660041 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.660011    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:55.660152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.660067    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:55.667275 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.667201    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.669204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.669093    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:55.669204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.669177    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.670091 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.669934    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.676805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.676762    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.677193 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.677023    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:21:55.677939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.677921    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"
Nov 03 16:21:55.679190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.677978    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm" event={"ID":"e8a6b2b1-84b1-43df-8495-85c3d904a0d7","Type":"ContainerDied","Data":"b29b1d390eb08ff77c7deef6773ce6e60a8a337c3de06c1e69ff045b9f5257da"}
Nov 03 16:21:55.679190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.678156    4964 scope.go:122] "RemoveContainer" containerID="d514fc8d7a0d69b6be01137bfeb591437c97582ac7bcb40f8492f2ad3fc0243d"
Nov 03 16:21:55.688822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.688617    4964 generic.go:356] "Generic (PLEG): container finished" podID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerID="21488714470cf06a5d0b4c54dec18245c5c871ca982ec5358b82fef7de96c02e" exitCode=0
Nov 03 16:21:55.688822 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.688769    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" event={"ID":"ecf81471-76cc-4465-8a94-bfdd83b764cb","Type":"ContainerDied","Data":"21488714470cf06a5d0b4c54dec18245c5c871ca982ec5358b82fef7de96c02e"}
Nov 03 16:21:55.691614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.690864    4964 generic.go:356] "Generic (PLEG): container finished" podID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerID="0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d" exitCode=0
Nov 03 16:21:55.691614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.690936    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" event={"ID":"077b8c38-65b7-443c-afe5-8d43d20f0a40","Type":"ContainerDied","Data":"0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d"}
Nov 03 16:21:55.691614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.690981    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54" event={"ID":"077b8c38-65b7-443c-afe5-8d43d20f0a40","Type":"ContainerDied","Data":"ef7ac8b02fdc6b72e6319e3dc83d6f98f824e0c841e491aaf146bb186a9c74dd"}
Nov 03 16:21:55.691614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.691058    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"
Nov 03 16:21:55.696413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.693698    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd" event={"ID":"1d4d726f-ef35-4781-8c27-431dad4ab68a","Type":"ContainerDied","Data":"6fd42d41ad633b4a9e55b0b176f0d5db36f1d109a7917d0ed326ebf7f3d0856e"}
Nov 03 16:21:55.696413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.693791    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"
Nov 03 16:21:55.726613 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.726578    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "kube-api-access-cskxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.744682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.736775    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token" pod "077b8c38-65b7-443c-afe5-8d43d20f0a40" (UID: "077b8c38-65b7-443c-afe5-8d43d20f0a40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:55.754628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.747257    4964 generic.go:356] "Generic (PLEG): container finished" podID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerID="34812bf92e76f534571ada6c536e7f506276400aacacff72cea2519a09ae389a" exitCode=0
Nov 03 16:21:55.754628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.747354    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" event={"ID":"e58b8929-76d9-4bf5-89d8-a8904d1c3366","Type":"ContainerDied","Data":"34812bf92e76f534571ada6c536e7f506276400aacacff72cea2519a09ae389a"}
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.757960    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.757995    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cskxj\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-kube-api-access-cskxj\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758010    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758024    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077b8c38-65b7-443c-afe5-8d43d20f0a40-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758037    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758052    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758065    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758079    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077b8c38-65b7-443c-afe5-8d43d20f0a40-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.758346 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.758091    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077b8c38-65b7-443c-afe5-8d43d20f0a40-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:55.771752 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.764813    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"]
Nov 03 16:21:55.789539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.787164    4964 generic.go:356] "Generic (PLEG): container finished" podID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerID="65023ca49df9f0bf7f11daf40c81b27bef29809bc4b7392f50dc050d8e55e705" exitCode=0
Nov 03 16:21:55.789539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.787997    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" event={"ID":"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df","Type":"ContainerDied","Data":"65023ca49df9f0bf7f11daf40c81b27bef29809bc4b7392f50dc050d8e55e705"}
Nov 03 16:21:55.794476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.794279    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:55.796873 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.796820    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:21:55.798355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.798312    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"]
Nov 03 16:21:55.799239 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.798953    4964 scope.go:122] "RemoveContainer" containerID="0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d"
Nov 03 16:21:55.806637 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.806573    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-2-w6-afb990-kserve-5b6b488987-h2kfm"]
Nov 03 16:21:55.881069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.879364    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"]
Nov 03 16:21:55.885357 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.885327    4964 scope.go:122] "RemoveContainer" containerID="0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d"
Nov 03 16:21:55.895327 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:21:55.894665    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d\": container with ID starting with 0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d not found: ID does not exist" containerID="0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d"
Nov 03 16:21:55.895327 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.894710    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d"} err="failed to get container status \"0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d\": rpc error: code = NotFound desc = could not find container \"0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d\": container with ID starting with 0cc93ad23bffd76bb87872e46fb508472f0408ff67cf47b559b6380f72cdb56d not found: ID does not exist"
Nov 03 16:21:55.895327 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.894739    4964 scope.go:122] "RemoveContainer" containerID="9403a1f5f6316265e3c78d8ec04c788a04839f6bef2c82bd7b0766031a1b8a84"
Nov 03 16:21:55.895327 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.895153    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-tkcpd"]
Nov 03 16:21:55.907697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.907667    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"]
Nov 03 16:21:55.911500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.910129    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:55.963294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.963252    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home\") pod \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") "
Nov 03 16:21:55.963680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.963654    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache\") pod \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") "
Nov 03 16:21:55.963876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.963844    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs\" (UniqueName: \"kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs\") pod \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") "
Nov 03 16:21:55.964059 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.964030    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.964281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.964266    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir\") pod \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") "
Nov 03 16:21:55.964457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.964432    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.964647 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.964618    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.964802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.964777    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.964970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.964954    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27\" (UniqueName: \"kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27\") pod \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") "
Nov 03 16:21:55.965098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.965077    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.965227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.965214    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm\") pod \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\" (UID: \"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df\") "
Nov 03 16:21:55.965323 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.965310    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.965448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.965434    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.965576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.965563    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.965783 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.965767    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket\") pod \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\" (UID: \"e58b8929-76d9-4bf5-89d8-a8904d1c3366\") "
Nov 03 16:21:55.967906 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.963693    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home" pod "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" (UID: "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.968040 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.967683    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache" pod "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" (UID: "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.968130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.967876    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.969333 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.969306    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.969671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.969643    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:55.970214 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.970187    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.970439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.970413    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir" pod "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" (UID: "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:55.970734 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:55.970706    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.009962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.009350    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm" pod "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" (UID: "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.009962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.009478    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.011896 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.011849    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs" pod "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" (UID: "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:56.013060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.013030    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:56.014278 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.014249    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27" pod "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" (UID: "da17a946-38b8-42c9-b3c8-fa6f7e8ec5df"). InnerVolumeSpecName "kube-api-access-ghv27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:56.015140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.015113    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "kube-api-access-2hgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:56.015263 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.015229    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo" pod "e58b8929-76d9-4bf5-89d8-a8904d1c3366" (UID: "e58b8929-76d9-4bf5-89d8-a8904d1c3366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:21:56.031372 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.031331    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"]
Nov 03 16:21:56.040479 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.040449    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-jxj54"]
Nov 03 16:21:56.068314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.068274    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir\") pod \"ecf81471-76cc-4465-8a94-bfdd83b764cb\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") "
Nov 03 16:21:56.068480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.068460    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2\" (UniqueName: \"kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2\") pod \"ecf81471-76cc-4465-8a94-bfdd83b764cb\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") "
Nov 03 16:21:56.068607 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.068592    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache\") pod \"ecf81471-76cc-4465-8a94-bfdd83b764cb\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") "
Nov 03 16:21:56.068654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.068637    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs\") pod \"ecf81471-76cc-4465-8a94-bfdd83b764cb\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") "
Nov 03 16:21:56.068715 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.068702    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm\") pod \"ecf81471-76cc-4465-8a94-bfdd83b764cb\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") "
Nov 03 16:21:56.068777 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.068763    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home\") pod \"ecf81471-76cc-4465-8a94-bfdd83b764cb\" (UID: \"ecf81471-76cc-4465-8a94-bfdd83b764cb\") "
Nov 03 16:21:56.069366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069346    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069376    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069407    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069421    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069440 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069435    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069449    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069466    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069480    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghv27\" (UniqueName: \"kubernetes.io/projected/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-kube-api-access-ghv27\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069497    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069511    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069525    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hgxv\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-kube-api-access-2hgxv\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069538    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069559 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069552    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b8929-76d9-4bf5-89d8-a8904d1c3366-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069566    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b8929-76d9-4bf5-89d8-a8904d1c3366-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.069776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.069582    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.070026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.070003    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home" pod "ecf81471-76cc-4465-8a94-bfdd83b764cb" (UID: "ecf81471-76cc-4465-8a94-bfdd83b764cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.070229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.070207    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir" pod "ecf81471-76cc-4465-8a94-bfdd83b764cb" (UID: "ecf81471-76cc-4465-8a94-bfdd83b764cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.071360 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.071100    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache" pod "ecf81471-76cc-4465-8a94-bfdd83b764cb" (UID: "ecf81471-76cc-4465-8a94-bfdd83b764cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.073065 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.073035    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs" pod "ecf81471-76cc-4465-8a94-bfdd83b764cb" (UID: "ecf81471-76cc-4465-8a94-bfdd83b764cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:56.073558 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.073494    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm" pod "ecf81471-76cc-4465-8a94-bfdd83b764cb" (UID: "ecf81471-76cc-4465-8a94-bfdd83b764cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:21:56.073658 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.073560    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2" pod "ecf81471-76cc-4465-8a94-bfdd83b764cb" (UID: "ecf81471-76cc-4465-8a94-bfdd83b764cb"). InnerVolumeSpecName "kube-api-access-7blx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:56.086178 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086154    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"]
Nov 03 16:21:56.086872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086851    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086873    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086890    4964 cpu_manager.go:414] "containerMap: removing container" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086897    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086928    4964 cpu_manager.go:414] "containerMap: removing container" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086937    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086951    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main"
Nov 03 16:21:56.086964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086957    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086967    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086974    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086985    4964 cpu_manager.go:414] "containerMap: removing container" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.086991    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087009    4964 cpu_manager.go:414] "containerMap: removing container" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087016    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087027    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087034    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087050    4964 cpu_manager.go:414] "containerMap: removing container" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087057    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087210    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087223    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087235    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087242    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" containerName="istio-proxy"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087254    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087264    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" containerName="istio-proxy"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087274    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087291    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" containerName="main"
Nov 03 16:21:56.087295 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.087300    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" containerName="istio-proxy"
Nov 03 16:21:56.092765 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.092749    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.107836 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.107806    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"]
Nov 03 16:21:56.171717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.171622    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blx2\" (UniqueName: \"kubernetes.io/projected/ecf81471-76cc-4465-8a94-bfdd83b764cb-kube-api-access-7blx2\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.171717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.171669    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.171717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.171689    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf81471-76cc-4465-8a94-bfdd83b764cb-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.171717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.171703    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.171717 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.171718    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.171982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.171731    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf81471-76cc-4465-8a94-bfdd83b764cb-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:56.272590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.272561    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.272745 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.272596    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflrz\" (UniqueName: \"kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.272794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.272775    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.374242 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.374200    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.374430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.374412    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.374483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.374455    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrz\" (UniqueName: \"kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.375053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.375028    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.376655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.376627    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.390639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.390613    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflrz\" (UniqueName: \"kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz\") pod \"maas-api-revoke-sub-a0227261-54zfk\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.426803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.426715    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:21:56.477889 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.477851    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 16:21:56.478196 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.477910    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 16:21:56.571320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.570981    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:56.571320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.571092    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:56.589462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.589133    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:21:56.589462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.589196    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:21:56.616539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.616504    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077b8c38-65b7-443c-afe5-8d43d20f0a40" path="/var/lib/kubelet/pods/077b8c38-65b7-443c-afe5-8d43d20f0a40/volumes"
Nov 03 16:21:56.617518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.617496    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4d726f-ef35-4781-8c27-431dad4ab68a" path="/var/lib/kubelet/pods/1d4d726f-ef35-4781-8c27-431dad4ab68a/volumes"
Nov 03 16:21:56.619216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.619194    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300c6366-5c06-4010-a4de-313fe2680332" path="/var/lib/kubelet/pods/300c6366-5c06-4010-a4de-313fe2680332/volumes"
Nov 03 16:21:56.620406 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.620372    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e31b7d-fda1-45b9-84b2-4295406c3357" path="/var/lib/kubelet/pods/44e31b7d-fda1-45b9-84b2-4295406c3357/volumes"
Nov 03 16:21:56.621122 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.621097    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c8edbf-14c3-4af7-8f15-e0b4b919b625" path="/var/lib/kubelet/pods/98c8edbf-14c3-4af7-8f15-e0b4b919b625/volumes"
Nov 03 16:21:56.621850 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.621830    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89127b8-55cb-4686-9d16-7009d7cca6be" path="/var/lib/kubelet/pods/b89127b8-55cb-4686-9d16-7009d7cca6be/volumes"
Nov 03 16:21:56.622987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.622968    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caec860b-340b-4777-9b61-b525bb9c42f6" path="/var/lib/kubelet/pods/caec860b-340b-4777-9b61-b525bb9c42f6/volumes"
Nov 03 16:21:56.623718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.623699    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a6b2b1-84b1-43df-8495-85c3d904a0d7" path="/var/lib/kubelet/pods/e8a6b2b1-84b1-43df-8495-85c3d904a0d7/volumes"
Nov 03 16:21:56.743901 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:21:56.743865    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d4caff_1075_4f3b_903b_8a9ac091fff2.slice/crio-1cbc9115fe73fe858ee887245f43b94198e9a006cda9dde3ba85b02089358abf WatchSource:0}: Error finding container 1cbc9115fe73fe858ee887245f43b94198e9a006cda9dde3ba85b02089358abf: Status 404 returned error can't find the container with id 1cbc9115fe73fe858ee887245f43b94198e9a006cda9dde3ba85b02089358abf
Nov 03 16:21:56.745774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.745742    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"]
Nov 03 16:21:56.801132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.800552    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc" event={"ID":"e58b8929-76d9-4bf5-89d8-a8904d1c3366","Type":"ContainerDied","Data":"841959e190f2a2daeb5299ea9af1cb0c7d38cac90cfa29cd47cf1caef6312f0f"}
Nov 03 16:21:56.801132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.800608    4964 scope.go:122] "RemoveContainer" containerID="34812bf92e76f534571ada6c536e7f506276400aacacff72cea2519a09ae389a"
Nov 03 16:21:56.801132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.800716    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"
Nov 03 16:21:56.808785 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.808685    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk" event={"ID":"06d4caff-1075-4f3b-903b-8a9ac091fff2","Type":"ContainerStarted","Data":"1cbc9115fe73fe858ee887245f43b94198e9a006cda9dde3ba85b02089358abf"}
Nov 03 16:21:56.811412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.810861    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2" event={"ID":"39957cfb-dda9-48dc-8374-aeacffe73815","Type":"ContainerStarted","Data":"7e97a1849b5e418d8be01e39b8fa283b3da5fe513a6f2dadddb720df6ca8ecbb"}
Nov 03 16:21:56.811412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.810914    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2" event={"ID":"39957cfb-dda9-48dc-8374-aeacffe73815","Type":"ContainerStarted","Data":"484b54a93c7a77c3398a9fb87385149a1b2d9f3cac7fbeb22f9290f587c03784"}
Nov 03 16:21:56.815078 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.814289    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7" event={"ID":"da17a946-38b8-42c9-b3c8-fa6f7e8ec5df","Type":"ContainerDied","Data":"d43a9029485bbb18bfbc13408b497673c912d73eaf4b016ab15954b95eb53212"}
Nov 03 16:21:56.815078 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.814407    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"
Nov 03 16:21:56.820581 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.819767    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7" event={"ID":"ecf81471-76cc-4465-8a94-bfdd83b764cb","Type":"ContainerDied","Data":"ed916fab157eb34a22b33bbabed3337591e8f04eea5a7667c31ee573a4cc9c18"}
Nov 03 16:21:56.820581 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.819853    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"
Nov 03 16:21:56.826088 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.825295    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5" event={"ID":"c4fb9651-3d02-461c-937a-20eb9241afdb","Type":"ContainerStarted","Data":"fd7a5901a9564e53b06d709492f2c4116133606579acee8f81851d0f1a29d6ac"}
Nov 03 16:21:56.826088 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.825347    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5" event={"ID":"c4fb9651-3d02-461c-937a-20eb9241afdb","Type":"ContainerStarted","Data":"8d774de4fe242774437b31f974ae3613dbdf4b26d119f7d2cce129b996a72925"}
Nov 03 16:21:56.852078 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.851956    4964 scope.go:122] "RemoveContainer" containerID="65023ca49df9f0bf7f11daf40c81b27bef29809bc4b7392f50dc050d8e55e705"
Nov 03 16:21:56.857905 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.857571    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2" podStartSLOduration=4.857535429 podStartE2EDuration="4.857535429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:56.847105612 +0000 UTC m=+1178.360487389" watchObservedRunningTime="2026-11-03 16:21:56.857535429 +0000 UTC m=+1178.370917207"
Nov 03 16:21:56.869682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.869655    4964 scope.go:122] "RemoveContainer" containerID="21488714470cf06a5d0b4c54dec18245c5c871ca982ec5358b82fef7de96c02e"
Nov 03 16:21:56.876579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.876521    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5" podStartSLOduration=5.876500725 podStartE2EDuration="5.876500725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:56.871190212 +0000 UTC m=+1178.384571990" watchObservedRunningTime="2026-11-03 16:21:56.876500725 +0000 UTC m=+1178.389882499"
Nov 03 16:21:56.894032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.893964    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"]
Nov 03 16:21:56.899141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.899103    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/simulator-w5-69cedd-kserve-556f678db7-fjfm7"]
Nov 03 16:21:56.913093 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.913051    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"]
Nov 03 16:21:56.922678 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.922647    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-w5-69cedd-kserve-c8b7bbccc-79kg7"]
Nov 03 16:21:56.941043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.941008    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"]
Nov 03 16:21:56.981548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:56.981502    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-kzbqc"]
Nov 03 16:21:57.594052 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.593967    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:57.595486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.595461    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:21:57.841707 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.841660    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk" event={"ID":"06d4caff-1075-4f3b-903b-8a9ac091fff2","Type":"ContainerStarted","Data":"96e2ef58dbffe6892482f17ba5121c01ecb14fbfd2cd164182a1b7cc7d9e4f4e"}
Nov 03 16:21:57.844977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.844891    4964 generic.go:356] "Generic (PLEG): container finished" podID="39957cfb-dda9-48dc-8374-aeacffe73815" containerID="7e97a1849b5e418d8be01e39b8fa283b3da5fe513a6f2dadddb720df6ca8ecbb" exitCode=0
Nov 03 16:21:57.845104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.845041    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2" event={"ID":"39957cfb-dda9-48dc-8374-aeacffe73815","Type":"ContainerDied","Data":"7e97a1849b5e418d8be01e39b8fa283b3da5fe513a6f2dadddb720df6ca8ecbb"}
Nov 03 16:21:57.852736 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.852700    4964 generic.go:356] "Generic (PLEG): container finished" podID="c4fb9651-3d02-461c-937a-20eb9241afdb" containerID="fd7a5901a9564e53b06d709492f2c4116133606579acee8f81851d0f1a29d6ac" exitCode=0
Nov 03 16:21:57.852897 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.852836    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5" event={"ID":"c4fb9651-3d02-461c-937a-20eb9241afdb","Type":"ContainerDied","Data":"fd7a5901a9564e53b06d709492f2c4116133606579acee8f81851d0f1a29d6ac"}
Nov 03 16:21:57.870338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:57.870277    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk" podStartSLOduration=1.870264812 podStartE2EDuration="1.870264812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:57.869353291 +0000 UTC m=+1179.382735068" watchObservedRunningTime="2026-11-03 16:21:57.870264812 +0000 UTC m=+1179.383646588"
Nov 03 16:21:58.520770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.520641    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:21:58.521222 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.521172    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:21:58.614187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.614148    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da17a946-38b8-42c9-b3c8-fa6f7e8ec5df" path="/var/lib/kubelet/pods/da17a946-38b8-42c9-b3c8-fa6f7e8ec5df/volumes"
Nov 03 16:21:58.615103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.615075    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58b8929-76d9-4bf5-89d8-a8904d1c3366" path="/var/lib/kubelet/pods/e58b8929-76d9-4bf5-89d8-a8904d1c3366/volumes"
Nov 03 16:21:58.615900 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.615857    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf81471-76cc-4465-8a94-bfdd83b764cb" path="/var/lib/kubelet/pods/ecf81471-76cc-4465-8a94-bfdd83b764cb/volumes"
Nov 03 16:21:58.641442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.641373    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:58.647928 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.647898    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:21:58.866224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.866097    4964 generic.go:356] "Generic (PLEG): container finished" podID="06d4caff-1075-4f3b-903b-8a9ac091fff2" containerID="96e2ef58dbffe6892482f17ba5121c01ecb14fbfd2cd164182a1b7cc7d9e4f4e" exitCode=0
Nov 03 16:21:58.866224 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:58.866173    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk" event={"ID":"06d4caff-1075-4f3b-903b-8a9ac091fff2","Type":"ContainerDied","Data":"96e2ef58dbffe6892482f17ba5121c01ecb14fbfd2cd164182a1b7cc7d9e4f4e"}
Nov 03 16:21:59.151158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.151128    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:59.153344 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.153322    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:59.221106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221071    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv\" (UniqueName: \"kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv\") pod \"39957cfb-dda9-48dc-8374-aeacffe73815\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") "
Nov 03 16:21:59.221253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221208    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca\") pod \"c4fb9651-3d02-461c-937a-20eb9241afdb\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") "
Nov 03 16:21:59.221290 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221261    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca\") pod \"39957cfb-dda9-48dc-8374-aeacffe73815\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") "
Nov 03 16:21:59.221801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221417    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s\" (UniqueName: \"kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s\") pod \"c4fb9651-3d02-461c-937a-20eb9241afdb\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") "
Nov 03 16:21:59.221801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221506    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca\") pod \"39957cfb-dda9-48dc-8374-aeacffe73815\" (UID: \"39957cfb-dda9-48dc-8374-aeacffe73815\") "
Nov 03 16:21:59.221801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221539    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca\") pod \"c4fb9651-3d02-461c-937a-20eb9241afdb\" (UID: \"c4fb9651-3d02-461c-937a-20eb9241afdb\") "
Nov 03 16:21:59.221801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221676    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca" pod "c4fb9651-3d02-461c-937a-20eb9241afdb" (UID: "c4fb9651-3d02-461c-937a-20eb9241afdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:59.222053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.221991    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca" pod "39957cfb-dda9-48dc-8374-aeacffe73815" (UID: "39957cfb-dda9-48dc-8374-aeacffe73815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:21:59.222155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.222130    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.222312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.222294    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.225739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.225530    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca" pod "c4fb9651-3d02-461c-937a-20eb9241afdb" (UID: "c4fb9651-3d02-461c-937a-20eb9241afdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:59.225739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.225643    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv" pod "39957cfb-dda9-48dc-8374-aeacffe73815" (UID: "39957cfb-dda9-48dc-8374-aeacffe73815"). InnerVolumeSpecName "kube-api-access-qd9xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:59.225739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.225707    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s" pod "c4fb9651-3d02-461c-937a-20eb9241afdb" (UID: "c4fb9651-3d02-461c-937a-20eb9241afdb"). InnerVolumeSpecName "kube-api-access-8qm8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:59.226097 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.226063    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca" pod "39957cfb-dda9-48dc-8374-aeacffe73815" (UID: "39957cfb-dda9-48dc-8374-aeacffe73815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:59.323834 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.323791    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qm8s\" (UniqueName: \"kubernetes.io/projected/c4fb9651-3d02-461c-937a-20eb9241afdb-kube-api-access-8qm8s\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.323834 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.323834    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fb9651-3d02-461c-937a-20eb9241afdb-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.324092 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.323850    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9xv\" (UniqueName: \"kubernetes.io/projected/39957cfb-dda9-48dc-8374-aeacffe73815-kube-api-access-qd9xv\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.324092 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.323862    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39957cfb-dda9-48dc-8374-aeacffe73815-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:59.876977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.876946    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"
Nov 03 16:21:59.876977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.876959    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5" event={"ID":"c4fb9651-3d02-461c-937a-20eb9241afdb","Type":"ContainerDied","Data":"8d774de4fe242774437b31f974ae3613dbdf4b26d119f7d2cce129b996a72925"}
Nov 03 16:21:59.877505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.877022    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d774de4fe242774437b31f974ae3613dbdf4b26d119f7d2cce129b996a72925"
Nov 03 16:21:59.883089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.878878    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"
Nov 03 16:21:59.883089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.878897    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2" event={"ID":"39957cfb-dda9-48dc-8374-aeacffe73815","Type":"ContainerDied","Data":"484b54a93c7a77c3398a9fb87385149a1b2d9f3cac7fbeb22f9290f587c03784"}
Nov 03 16:21:59.883089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:21:59.878937    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="484b54a93c7a77c3398a9fb87385149a1b2d9f3cac7fbeb22f9290f587c03784"
Nov 03 16:22:00.067483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.067452    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:22:00.135540 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.135451    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca\") pod \"06d4caff-1075-4f3b-903b-8a9ac091fff2\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") "
Nov 03 16:22:00.135682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.135549    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca\") pod \"06d4caff-1075-4f3b-903b-8a9ac091fff2\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") "
Nov 03 16:22:00.135682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.135639    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz\" (UniqueName: \"kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz\") pod \"06d4caff-1075-4f3b-903b-8a9ac091fff2\" (UID: \"06d4caff-1075-4f3b-903b-8a9ac091fff2\") "
Nov 03 16:22:00.135912 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.135883    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca" pod "06d4caff-1075-4f3b-903b-8a9ac091fff2" (UID: "06d4caff-1075-4f3b-903b-8a9ac091fff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:00.136233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.136207    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:00.137805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.137771    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca" pod "06d4caff-1075-4f3b-903b-8a9ac091fff2" (UID: "06d4caff-1075-4f3b-903b-8a9ac091fff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:00.137892 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.137862    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz" pod "06d4caff-1075-4f3b-903b-8a9ac091fff2" (UID: "06d4caff-1075-4f3b-903b-8a9ac091fff2"). InnerVolumeSpecName "kube-api-access-xflrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:00.237903 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.237865    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d4caff-1075-4f3b-903b-8a9ac091fff2-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:00.237903 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.237896    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflrz\" (UniqueName: \"kubernetes.io/projected/06d4caff-1075-4f3b-903b-8a9ac091fff2-kube-api-access-xflrz\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:00.888451 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.888324    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"
Nov 03 16:22:00.888451 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.888332    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk" event={"ID":"06d4caff-1075-4f3b-903b-8a9ac091fff2","Type":"ContainerDied","Data":"1cbc9115fe73fe858ee887245f43b94198e9a006cda9dde3ba85b02089358abf"}
Nov 03 16:22:00.888881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:00.888465    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cbc9115fe73fe858ee887245f43b94198e9a006cda9dde3ba85b02089358abf"
Nov 03 16:22:01.879095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.879060    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:01.879453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.879245    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:01.879958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.879931    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:01.920466 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.920414    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"]
Nov 03 16:22:01.921187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921167    4964 cpu_manager.go:414] "containerMap: removing container" podUID="06d4caff-1075-4f3b-903b-8a9ac091fff2" containerName="revoke-keys"
Nov 03 16:22:01.921187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921185    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d4caff-1075-4f3b-903b-8a9ac091fff2" containerName="revoke-keys"
Nov 03 16:22:01.921338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921212    4964 cpu_manager.go:414] "containerMap: removing container" podUID="39957cfb-dda9-48dc-8374-aeacffe73815" containerName="revoke-keys"
Nov 03 16:22:01.921338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921218    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39957cfb-dda9-48dc-8374-aeacffe73815" containerName="revoke-keys"
Nov 03 16:22:01.921338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921235    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c4fb9651-3d02-461c-937a-20eb9241afdb" containerName="revoke-keys"
Nov 03 16:22:01.921338 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921241    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fb9651-3d02-461c-937a-20eb9241afdb" containerName="revoke-keys"
Nov 03 16:22:01.921568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921343    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="39957cfb-dda9-48dc-8374-aeacffe73815" containerName="revoke-keys"
Nov 03 16:22:01.921568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921353    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fb9651-3d02-461c-937a-20eb9241afdb" containerName="revoke-keys"
Nov 03 16:22:01.921568 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.921364    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d4caff-1075-4f3b-903b-8a9ac091fff2" containerName="revoke-keys"
Nov 03 16:22:01.925172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.925148    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:01.931448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.931419    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:22:01.941409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.933159    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:22:01.947023 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:01.946984    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"]
Nov 03 16:22:02.002345 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.002307    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdp8\" (UniqueName: \"kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.002739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.002699    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.003094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.003076    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.104903 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.104863    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdp8\" (UniqueName: \"kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.104903 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.104913    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.105184 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.105068    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.105796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.105769    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.108156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.108134    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.118995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.118963    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdp8\" (UniqueName: \"kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8\") pod \"maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.266297 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.266264    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:02.272036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.271966    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"]
Nov 03 16:22:02.273293 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.273262    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" containerID="cri-o://18fd0293c44e884aa41416961a4f18750aabf623ef33fdf3e6258a49bd4b8700" gracePeriod=30
Nov 03 16:22:02.288428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.280245    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"]
Nov 03 16:22:02.288428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.280541    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" containerID="cri-o://22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb" gracePeriod=30
Nov 03 16:22:02.669636 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:02.668589    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:22:02.816773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.816654    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"]
Nov 03 16:22:02.911155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:02.911113    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4" event={"ID":"64288cf0-1310-4f63-aa0f-cc124c8b3bff","Type":"ContainerStarted","Data":"9cf40d2857e4b3a6d355f268b0733bfb8b4082009ba931440176e796ef1b59c6"}
Nov 03 16:22:03.223587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:03.223540    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:03.224201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:03.224155    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:03.935259 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:03.935222    4964 generic.go:356] "Generic (PLEG): container finished" podID="64288cf0-1310-4f63-aa0f-cc124c8b3bff" containerID="a8b71cf6b8655487b3c030107c8adc0ceca481366062acf778d5ad44f59f2d97" exitCode=0
Nov 03 16:22:03.935476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:03.935306    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4" event={"ID":"64288cf0-1310-4f63-aa0f-cc124c8b3bff","Type":"ContainerDied","Data":"a8b71cf6b8655487b3c030107c8adc0ceca481366062acf778d5ad44f59f2d97"}
Nov 03 16:22:05.414870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.414843    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:05.433342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.432187    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8\" (UniqueName: \"kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8\") pod \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") "
Nov 03 16:22:05.433342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.432432    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca\") pod \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") "
Nov 03 16:22:05.433342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.432504    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca\") pod \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\" (UID: \"64288cf0-1310-4f63-aa0f-cc124c8b3bff\") "
Nov 03 16:22:05.433342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.432995    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca" pod "64288cf0-1310-4f63-aa0f-cc124c8b3bff" (UID: "64288cf0-1310-4f63-aa0f-cc124c8b3bff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:05.434743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.434714    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca" pod "64288cf0-1310-4f63-aa0f-cc124c8b3bff" (UID: "64288cf0-1310-4f63-aa0f-cc124c8b3bff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:05.434931 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.434911    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8" pod "64288cf0-1310-4f63-aa0f-cc124c8b3bff" (UID: "64288cf0-1310-4f63-aa0f-cc124c8b3bff"). InnerVolumeSpecName "kube-api-access-mqdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:05.543809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.543671    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdp8\" (UniqueName: \"kubernetes.io/projected/64288cf0-1310-4f63-aa0f-cc124c8b3bff-kube-api-access-mqdp8\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.543809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.543711    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.543809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.543728    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64288cf0-1310-4f63-aa0f-cc124c8b3bff-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:05.956398 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.956352    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"
Nov 03 16:22:05.956601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.956402    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4" event={"ID":"64288cf0-1310-4f63-aa0f-cc124c8b3bff","Type":"ContainerDied","Data":"9cf40d2857e4b3a6d355f268b0733bfb8b4082009ba931440176e796ef1b59c6"}
Nov 03 16:22:05.956601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:05.956465    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cf40d2857e4b3a6d355f268b0733bfb8b4082009ba931440176e796ef1b59c6"
Nov 03 16:22:06.254966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.254887    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:06.254966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.254953    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:06.550159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.549822    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"]
Nov 03 16:22:06.847741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.847290    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"]
Nov 03 16:22:06.847741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.847611    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy" containerID="cri-o://50d56c120960c4a36ea361d3a5bb24e5055124e838684f4ac27df5633c313a84" gracePeriod=30
Nov 03 16:22:06.944498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.944466    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:22:06.967381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.967346    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume\") pod \"923d75e2-cd92-48de-9284-6e159e0a359a\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") "
Nov 03 16:22:06.967595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.967474    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p\" (UniqueName: \"kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p\") pod \"923d75e2-cd92-48de-9284-6e159e0a359a\" (UID: \"923d75e2-cd92-48de-9284-6e159e0a359a\") "
Nov 03 16:22:06.968832 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.968715    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume" pod "923d75e2-cd92-48de-9284-6e159e0a359a" (UID: "923d75e2-cd92-48de-9284-6e159e0a359a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:06.970032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.970008    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p" pod "923d75e2-cd92-48de-9284-6e159e0a359a" (UID: "923d75e2-cd92-48de-9284-6e159e0a359a"). InnerVolumeSpecName "kube-api-access-vfm4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:06.971256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.970665    4964 generic.go:356] "Generic (PLEG): container finished" podID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerID="abfbe3daaf93bd704c692b1768836f2ee684de5f103c8318e84d048e0d90dd92" exitCode=0
Nov 03 16:22:06.971256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.970740    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" event={"ID":"81ba2d14-53e6-4bfb-b8fe-0d057eb41768","Type":"ContainerDied","Data":"abfbe3daaf93bd704c692b1768836f2ee684de5f103c8318e84d048e0d90dd92"}
Nov 03 16:22:06.972119 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.972099    4964 generic.go:356] "Generic (PLEG): container finished" podID="923d75e2-cd92-48de-9284-6e159e0a359a" containerID="32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933" exitCode=0
Nov 03 16:22:06.972219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.972158    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"
Nov 03 16:22:06.972219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.972183    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" event={"ID":"923d75e2-cd92-48de-9284-6e159e0a359a","Type":"ContainerDied","Data":"32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933"}
Nov 03 16:22:06.972334 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.972236    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94" event={"ID":"923d75e2-cd92-48de-9284-6e159e0a359a","Type":"ContainerDied","Data":"2e61f7d884029ebb481dd32933f6bc8929f13516bb35e6a1ca386edfe645347e"}
Nov 03 16:22:06.972334 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.972268    4964 scope.go:122] "RemoveContainer" containerID="32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933"
Nov 03 16:22:06.987459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.987362    4964 scope.go:122] "RemoveContainer" containerID="32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933"
Nov 03 16:22:06.987862 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:06.987788    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933\": container with ID starting with 32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933 not found: ID does not exist" containerID="32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933"
Nov 03 16:22:06.987862 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:06.987827    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933"} err="failed to get container status \"32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933\": rpc error: code = NotFound desc = could not find container \"32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933\": container with ID starting with 32de7578c8494538daeb6da7b531d81df985f6ef15680f9e90130ae219ce1933 not found: ID does not exist"
Nov 03 16:22:07.009888 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.008626    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"]
Nov 03 16:22:07.023936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.023215    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-afb990-76497f4799-k8x94"]
Nov 03 16:22:07.074637 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.073591    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfm4p\" (UniqueName: \"kubernetes.io/projected/923d75e2-cd92-48de-9284-6e159e0a359a-kube-api-access-vfm4p\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.074637 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.073638    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923d75e2-cd92-48de-9284-6e159e0a359a-plugins-config-volume\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.086460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.084041    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:22:07.174247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.174211    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t\" (UniqueName: \"kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t\") pod \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") "
Nov 03 16:22:07.174446 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.174432    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume\") pod \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\" (UID: \"81ba2d14-53e6-4bfb-b8fe-0d057eb41768\") "
Nov 03 16:22:07.174890 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.174856    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume" pod "81ba2d14-53e6-4bfb-b8fe-0d057eb41768" (UID: "81ba2d14-53e6-4bfb-b8fe-0d057eb41768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:07.175009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.174986    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-plugins-config-volume\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.176547 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.176523    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t" pod "81ba2d14-53e6-4bfb-b8fe-0d057eb41768" (UID: "81ba2d14-53e6-4bfb-b8fe-0d057eb41768"). InnerVolumeSpecName "kube-api-access-jk28t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.276142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.276101    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk28t\" (UniqueName: \"kubernetes.io/projected/81ba2d14-53e6-4bfb-b8fe-0d057eb41768-kube-api-access-jk28t\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.536378 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.536349    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:22:07.586325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586291    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586364    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586451    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586486    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586594    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586679    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586709    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586767    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.586835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.586834    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo\") pod \"577d79ca-7173-4aee-9c28-88dd5300275a\" (UID: \"577d79ca-7173-4aee-9c28-88dd5300275a\") "
Nov 03 16:22:07.587512 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.587471    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.587512 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.587487    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.588025 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.587803    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.588025 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.587822    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.588025 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.587984    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:07.589826 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.589798    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:07.589990 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.589961    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:07.590460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.590431    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "kube-api-access-5p2n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.590833 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.590801    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token" pod "577d79ca-7173-4aee-9c28-88dd5300275a" (UID: "577d79ca-7173-4aee-9c28-88dd5300275a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:07.688343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688304    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688338    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2n4\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-kube-api-access-5p2n4\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688350    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688359    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688368    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688377    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577d79ca-7173-4aee-9c28-88dd5300275a-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688398    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577d79ca-7173-4aee-9c28-88dd5300275a-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688409    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577d79ca-7173-4aee-9c28-88dd5300275a-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.688616 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.688417    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577d79ca-7173-4aee-9c28-88dd5300275a-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:07.983933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.983458    4964 generic.go:356] "Generic (PLEG): container finished" podID="577d79ca-7173-4aee-9c28-88dd5300275a" containerID="22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb" exitCode=0
Nov 03 16:22:07.983933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.983506    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" event={"ID":"577d79ca-7173-4aee-9c28-88dd5300275a","Type":"ContainerDied","Data":"22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb"}
Nov 03 16:22:07.983933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.983534    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"
Nov 03 16:22:07.983933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.983586    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625" event={"ID":"577d79ca-7173-4aee-9c28-88dd5300275a","Type":"ContainerDied","Data":"c54f7a06d7ca02d4b9bbb35aa9cbf5ab46b6bab7882c382bb05346ea915b053d"}
Nov 03 16:22:07.983933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.983603    4964 scope.go:122] "RemoveContainer" containerID="22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb"
Nov 03 16:22:07.993898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.985829    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"
Nov 03 16:22:07.993898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.985850    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk" event={"ID":"81ba2d14-53e6-4bfb-b8fe-0d057eb41768","Type":"ContainerDied","Data":"cc3a6097ea9490cf979f221a225920c011c782a1d2108cd5b1e3776466e12599"}
Nov 03 16:22:07.993898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.989435    4964 generic.go:356] "Generic (PLEG): container finished" podID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerID="18fd0293c44e884aa41416961a4f18750aabf623ef33fdf3e6258a49bd4b8700" exitCode=0
Nov 03 16:22:07.993898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.989496    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" event={"ID":"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec","Type":"ContainerDied","Data":"18fd0293c44e884aa41416961a4f18750aabf623ef33fdf3e6258a49bd4b8700"}
Nov 03 16:22:07.993898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.989557    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s" event={"ID":"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec","Type":"ContainerDied","Data":"8da291c72949e7e0d13e52036f155a10cfb6a4edac4f29fdaa1f2d4c9da7235d"}
Nov 03 16:22:07.993898 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:07.989584    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da291c72949e7e0d13e52036f155a10cfb6a4edac4f29fdaa1f2d4c9da7235d"
Nov 03 16:22:08.012892 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.012805    4964 scope.go:122] "RemoveContainer" containerID="22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb"
Nov 03 16:22:08.013348 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:08.013315    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb\": container with ID starting with 22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb not found: ID does not exist" containerID="22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb"
Nov 03 16:22:08.013522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.013356    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb"} err="failed to get container status \"22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb\": rpc error: code = NotFound desc = could not find container \"22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb\": container with ID starting with 22d9ca6332e5486ce12a8f9ec6143c4c15024ac59bc427f0d6915353168555fb not found: ID does not exist"
Nov 03 16:22:08.013522 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.013381    4964 scope.go:122] "RemoveContainer" containerID="abfbe3daaf93bd704c692b1768836f2ee684de5f103c8318e84d048e0d90dd92"
Nov 03 16:22:08.015184 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.015166    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:22:08.029059 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.029034    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"]
Nov 03 16:22:08.037359 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.037330    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-g4625"]
Nov 03 16:22:08.050402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.050366    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"]
Nov 03 16:22:08.056239 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.056213    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-afb990-54654745-22rvk"]
Nov 03 16:22:08.092632 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092601    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.092827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092656    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.092827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092697    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.092827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092713    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.092827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092750    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.092827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092801    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.093096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092841    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.093096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092870    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.093096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.092958    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data\") pod \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\" (UID: \"027fe49a-1c99-41a6-bbd3-c5a324f9b4ec\") "
Nov 03 16:22:08.093241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093187    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:08.093241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093201    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:08.093462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093381    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:08.093462 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093410    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:08.093608 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093589    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.093665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093602    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:08.093665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093614    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.093665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093630    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.093665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.093644    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.096143 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.096116    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:08.096630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.096602    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:08.096861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.096840    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:08.097244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.097216    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c" pod "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" (UID: "027fe49a-1c99-41a6-bbd3-c5a324f9b4ec"). InnerVolumeSpecName "kube-api-access-97k2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:08.195114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.195078    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.195114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.195110    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97k2c\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-kube-api-access-97k2c\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.195114 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.195121    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.195355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.195131    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.195355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.195140    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:08.612529 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.612441    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" path="/var/lib/kubelet/pods/577d79ca-7173-4aee-9c28-88dd5300275a/volumes"
Nov 03 16:22:08.613021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.612868    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" path="/var/lib/kubelet/pods/81ba2d14-53e6-4bfb-b8fe-0d057eb41768/volumes"
Nov 03 16:22:08.613216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.613202    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" path="/var/lib/kubelet/pods/923d75e2-cd92-48de-9284-6e159e0a359a/volumes"
Nov 03 16:22:08.999273 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:08.999244    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"
Nov 03 16:22:09.022838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:09.022791    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"]
Nov 03 16:22:09.027509 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:09.027479    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-64m5s"]
Nov 03 16:22:10.628897 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:10.622380    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" path="/var/lib/kubelet/pods/027fe49a-1c99-41a6-bbd3-c5a324f9b4ec/volumes"
Nov 03 16:22:11.878814 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:11.878785    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:22:12.056631 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.054573    4964 generic.go:356] "Generic (PLEG): container finished" podID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerID="50d56c120960c4a36ea361d3a5bb24e5055124e838684f4ac27df5633c313a84" exitCode=0
Nov 03 16:22:12.056631 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.054774    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" event={"ID":"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb","Type":"ContainerDied","Data":"50d56c120960c4a36ea361d3a5bb24e5055124e838684f4ac27df5633c313a84"}
Nov 03 16:22:12.234094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.221721    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:22:12.390592 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390560    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.390786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390602    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.390786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390660    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.390786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390679    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.390786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390721    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.390786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390779    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.391053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390823    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.391053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390882    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.391053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390901    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.391053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390912    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket\") pod \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\" (UID: \"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb\") "
Nov 03 16:22:12.391053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.390963    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.391053 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.391009    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.391325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.391114    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.391325 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.391122    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:12.391672 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.391646    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.391835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.391817    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.391952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.391937    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.392038    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.392143 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.392131    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.393078 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.393050    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:12.393299 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.393282    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:12.393556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.393536    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:12.393617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.393556    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx" pod "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" (UID: "0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb"). InnerVolumeSpecName "kube-api-access-n86xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:12.493754 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.493662    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.493754 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.493702    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86xx\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-kube-api-access-n86xx\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.493754 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.493713    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:12.493754 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:12.493722    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:13.094879 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.094835    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v" event={"ID":"0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb","Type":"ContainerDied","Data":"fe255d47c88855a93bfddf40c3114bd5f3e27f83f0655ccdfd17a5bf70c2bbf5"}
Nov 03 16:22:13.095468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.094915    4964 scope.go:122] "RemoveContainer" containerID="50d56c120960c4a36ea361d3a5bb24e5055124e838684f4ac27df5633c313a84"
Nov 03 16:22:13.095468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.095138    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.102790    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"]
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110346    4964 cpu_manager.go:414] "containerMap: removing container" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110372    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110438    4964 cpu_manager.go:414] "containerMap: removing container" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110447    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110458    4964 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110466    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110490    4964 cpu_manager.go:414] "containerMap: removing container" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" containerName="payload-pre-processing"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110497    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" containerName="payload-pre-processing"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110522    4964 cpu_manager.go:414] "containerMap: removing container" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerName="payload-processing"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110529    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerName="payload-processing"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110545    4964 cpu_manager.go:414] "containerMap: removing container" podUID="64288cf0-1310-4f63-aa0f-cc124c8b3bff" containerName="revoke-keys"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110552    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64288cf0-1310-4f63-aa0f-cc124c8b3bff" containerName="revoke-keys"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110710    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="577d79ca-7173-4aee-9c28-88dd5300275a" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110724    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="027fe49a-1c99-41a6-bbd3-c5a324f9b4ec" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110733    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" containerName="istio-proxy"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110745    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="923d75e2-cd92-48de-9284-6e159e0a359a" containerName="payload-pre-processing"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110757    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ba2d14-53e6-4bfb-b8fe-0d057eb41768" containerName="payload-processing"
Nov 03 16:22:13.112804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.110768    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="64288cf0-1310-4f63-aa0f-cc124c8b3bff" containerName="revoke-keys"
Nov 03 16:22:13.119935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.119907    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.124690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.124572    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:22:13.124690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.124663    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:22:13.125687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.125664    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"]
Nov 03 16:22:13.177876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.177806    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"]
Nov 03 16:22:13.185129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.184986    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6bg5v"]
Nov 03 16:22:13.221159 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.221118    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.221547 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.221524    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv8p\" (UniqueName: \"kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.221750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.221730    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.325482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.324748    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.325482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.323974    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.325482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.325010    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv8p\" (UniqueName: \"kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.325482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.325433    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.329254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.329225    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.339352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.339247    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv8p\" (UniqueName: \"kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p\") pod \"maas-api-revoke-sub-46173954-57jtq\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.368730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.368701    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:22:13.486414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.478161    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:13.541536 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.540151    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume\") pod \"e21428f7-a0cc-4487-81a1-8a77a969eb50\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") "
Nov 03 16:22:13.541536 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.540442    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h\" (UniqueName: \"kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h\") pod \"e21428f7-a0cc-4487-81a1-8a77a969eb50\" (UID: \"e21428f7-a0cc-4487-81a1-8a77a969eb50\") "
Nov 03 16:22:13.542020 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.541959    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume" pod "e21428f7-a0cc-4487-81a1-8a77a969eb50" (UID: "e21428f7-a0cc-4487-81a1-8a77a969eb50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:13.552628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.552569    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h" pod "e21428f7-a0cc-4487-81a1-8a77a969eb50" (UID: "e21428f7-a0cc-4487-81a1-8a77a969eb50"). InnerVolumeSpecName "kube-api-access-4cn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:13.655899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.653091    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cn4h\" (UniqueName: \"kubernetes.io/projected/e21428f7-a0cc-4487-81a1-8a77a969eb50-kube-api-access-4cn4h\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:13.655899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.653135    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21428f7-a0cc-4487-81a1-8a77a969eb50-plugins-config-volume\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:13.848978 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:13.848941    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"]
Nov 03 16:22:13.854008 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:22:13.853977    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice/crio-3ae8687b0ed151b0208bc1df9c7bef1e12f1246ef4050f95ce546c9f15f97556 WatchSource:0}: Error finding container 3ae8687b0ed151b0208bc1df9c7bef1e12f1246ef4050f95ce546c9f15f97556: Status 404 returned error can't find the container with id 3ae8687b0ed151b0208bc1df9c7bef1e12f1246ef4050f95ce546c9f15f97556
Nov 03 16:22:14.065411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.062838    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"]
Nov 03 16:22:14.065411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.063124    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68" podUID="41c02edc-4f93-43ba-9648-6fb680df7d4f" containerName="maas-api" containerID="cri-o://6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4" gracePeriod=30
Nov 03 16:22:14.160982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.160942    4964 generic.go:356] "Generic (PLEG): container finished" podID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerID="703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6" exitCode=0
Nov 03 16:22:14.161899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.161653    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"
Nov 03 16:22:14.167041 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.161677    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" event={"ID":"e21428f7-a0cc-4487-81a1-8a77a969eb50","Type":"ContainerDied","Data":"703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6"}
Nov 03 16:22:14.167305 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.167278    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78" event={"ID":"e21428f7-a0cc-4487-81a1-8a77a969eb50","Type":"ContainerDied","Data":"4dac312c7f3d3d395357fd1058a5643f75db6cb64ce0587a8e8e0c8a3f0f9a52"}
Nov 03 16:22:14.168723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.167486    4964 scope.go:122] "RemoveContainer" containerID="703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6"
Nov 03 16:22:14.168918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.168895    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq" event={"ID":"844ce432-ab27-4dbc-9ea2-10f933b38198","Type":"ContainerStarted","Data":"3ae8687b0ed151b0208bc1df9c7bef1e12f1246ef4050f95ce546c9f15f97556"}
Nov 03 16:22:14.201070 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.200890    4964 scope.go:122] "RemoveContainer" containerID="703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6"
Nov 03 16:22:14.201271 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:14.201198    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6\": container with ID starting with 703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6 not found: ID does not exist" containerID="703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6"
Nov 03 16:22:14.201271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.201226    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6"} err="failed to get container status \"703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6\": rpc error: code = NotFound desc = could not find container \"703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6\": container with ID starting with 703d8a94a6bcd2f26c7998d8b558989987d0ed198e7c85a28ad777e0842b35c6 not found: ID does not exist"
Nov 03 16:22:14.257902 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.257858    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"]
Nov 03 16:22:14.267690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.267355    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-69cedd-57b99699f-p9p78"]
Nov 03 16:22:14.566912 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.566851    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:22:14.597886 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.597851    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2\" (UniqueName: \"kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2\") pod \"41c02edc-4f93-43ba-9648-6fb680df7d4f\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") "
Nov 03 16:22:14.598055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.597902    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs\") pod \"41c02edc-4f93-43ba-9648-6fb680df7d4f\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") "
Nov 03 16:22:14.598055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.597944    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls\") pod \"41c02edc-4f93-43ba-9648-6fb680df7d4f\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") "
Nov 03 16:22:14.598210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.598068    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca\") pod \"41c02edc-4f93-43ba-9648-6fb680df7d4f\" (UID: \"41c02edc-4f93-43ba-9648-6fb680df7d4f\") "
Nov 03 16:22:14.598705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.598676    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca" pod "41c02edc-4f93-43ba-9648-6fb680df7d4f" (UID: "41c02edc-4f93-43ba-9648-6fb680df7d4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:14.600802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.600773    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs" pod "41c02edc-4f93-43ba-9648-6fb680df7d4f" (UID: "41c02edc-4f93-43ba-9648-6fb680df7d4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:14.620781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.611982    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2" pod "41c02edc-4f93-43ba-9648-6fb680df7d4f" (UID: "41c02edc-4f93-43ba-9648-6fb680df7d4f"). InnerVolumeSpecName "kube-api-access-xw4m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:14.620781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.613416    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls" pod "41c02edc-4f93-43ba-9648-6fb680df7d4f" (UID: "41c02edc-4f93-43ba-9648-6fb680df7d4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:14.620781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.614612    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb" path="/var/lib/kubelet/pods/0e1c9f81-73b7-403d-88f9-8f6c0fa21fcb/volumes"
Nov 03 16:22:14.620781 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.615256    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" path="/var/lib/kubelet/pods/e21428f7-a0cc-4487-81a1-8a77a969eb50/volumes"
Nov 03 16:22:14.699798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.699702    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c02edc-4f93-43ba-9648-6fb680df7d4f-openshift-service-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:14.699798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.699739    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4m2\" (UniqueName: \"kubernetes.io/projected/41c02edc-4f93-43ba-9648-6fb680df7d4f-kube-api-access-xw4m2\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:14.699798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.699757    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-metrics-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:14.699798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.699773    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c02edc-4f93-43ba-9648-6fb680df7d4f-maas-api-tls\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:14.890827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.890791    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:22:14.930927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.930892    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"]
Nov 03 16:22:14.931317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.931279    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy" containerID="cri-o://ed27238aeedbf09944179c635daef09396382b1a8a717100fafd7d1343dc37bb" gracePeriod=30
Nov 03 16:22:14.969567 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.969528    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"]
Nov 03 16:22:14.969876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.969843    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" containerID="cri-o://a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842" gracePeriod=30
Nov 03 16:22:14.993710 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.993677    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"]
Nov 03 16:22:14.993934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:14.993909    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" containerID="cri-o://3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586" gracePeriod=30
Nov 03 16:22:15.182220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.182185    4964 generic.go:356] "Generic (PLEG): container finished" podID="844ce432-ab27-4dbc-9ea2-10f933b38198" containerID="ee22df8b2da17fb88678109fbe49a09b0d14f83efdd190d311fda50f2bb9f6b0" exitCode=0
Nov 03 16:22:15.182720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.182271    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq" event={"ID":"844ce432-ab27-4dbc-9ea2-10f933b38198","Type":"ContainerDied","Data":"ee22df8b2da17fb88678109fbe49a09b0d14f83efdd190d311fda50f2bb9f6b0"}
Nov 03 16:22:15.184030 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.184010    4964 generic.go:356] "Generic (PLEG): container finished" podID="41c02edc-4f93-43ba-9648-6fb680df7d4f" containerID="6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4" exitCode=0
Nov 03 16:22:15.184141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.184118    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"
Nov 03 16:22:15.184240 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.184120    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68" event={"ID":"41c02edc-4f93-43ba-9648-6fb680df7d4f","Type":"ContainerDied","Data":"6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4"}
Nov 03 16:22:15.184280 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.184264    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68" event={"ID":"41c02edc-4f93-43ba-9648-6fb680df7d4f","Type":"ContainerDied","Data":"892a3370a5c54a3ac6a8d382c0550e031dc3bf092bb9b3f99ab914164eb67a69"}
Nov 03 16:22:15.184329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.184298    4964 scope.go:122] "RemoveContainer" containerID="6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4"
Nov 03 16:22:15.196080 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.196051    4964 scope.go:122] "RemoveContainer" containerID="6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4"
Nov 03 16:22:15.196423 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:15.196375    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4\": container with ID starting with 6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4 not found: ID does not exist" containerID="6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4"
Nov 03 16:22:15.196518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.196427    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4"} err="failed to get container status \"6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4\": rpc error: code = NotFound desc = could not find container \"6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4\": container with ID starting with 6003c7e1834939b6ec94bbd7d32f071e34bf4168534b482a4d4d053876ecd2c4 not found: ID does not exist"
Nov 03 16:22:15.227790 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.227759    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"]
Nov 03 16:22:15.234574 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:15.234544    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-69cedd-f6bd58645-crt68"]
Nov 03 16:22:16.861856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:16.861816    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c02edc-4f93-43ba-9648-6fb680df7d4f" path="/var/lib/kubelet/pods/41c02edc-4f93-43ba-9648-6fb680df7d4f/volumes"
Nov 03 16:22:17.082848 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.082811    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"]
Nov 03 16:22:17.084029 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.083998    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerName="payload-pre-processing"
Nov 03 16:22:17.084190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.084174    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerName="payload-pre-processing"
Nov 03 16:22:17.084316 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.084302    4964 cpu_manager.go:414] "containerMap: removing container" podUID="41c02edc-4f93-43ba-9648-6fb680df7d4f" containerName="maas-api"
Nov 03 16:22:17.084415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.084401    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c02edc-4f93-43ba-9648-6fb680df7d4f" containerName="maas-api"
Nov 03 16:22:17.084672 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.084656    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21428f7-a0cc-4487-81a1-8a77a969eb50" containerName="payload-pre-processing"
Nov 03 16:22:17.084772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.084761    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c02edc-4f93-43ba-9648-6fb680df7d4f" containerName="maas-api"
Nov 03 16:22:17.154252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.154219    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.210439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.195661    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:17.210439 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.195721    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:17.211086 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.211037    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"]
Nov 03 16:22:17.287937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.287897    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.288186 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.288162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnhb\" (UniqueName: \"kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.288349 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.288331    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.496464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.480409    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.496464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.480488    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnhb\" (UniqueName: \"kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.496464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.480574    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.496464 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.489086    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.606418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.600332    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnhb\" (UniqueName: \"kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.606418 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.600962    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:17.854130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:17.854042    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:18.398826 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.372875    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:18.553654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.552626    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca\") pod \"844ce432-ab27-4dbc-9ea2-10f933b38198\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") "
Nov 03 16:22:18.553654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.552703    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p\" (UniqueName: \"kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p\") pod \"844ce432-ab27-4dbc-9ea2-10f933b38198\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") "
Nov 03 16:22:18.553654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.552856    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca\") pod \"844ce432-ab27-4dbc-9ea2-10f933b38198\" (UID: \"844ce432-ab27-4dbc-9ea2-10f933b38198\") "
Nov 03 16:22:18.638428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.569591    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca" pod "844ce432-ab27-4dbc-9ea2-10f933b38198" (UID: "844ce432-ab27-4dbc-9ea2-10f933b38198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:18.736595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.736535    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca" pod "844ce432-ab27-4dbc-9ea2-10f933b38198" (UID: "844ce432-ab27-4dbc-9ea2-10f933b38198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:18.803629 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.764720    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p" pod "844ce432-ab27-4dbc-9ea2-10f933b38198" (UID: "844ce432-ab27-4dbc-9ea2-10f933b38198"). InnerVolumeSpecName "kube-api-access-shv8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:18.906414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.899969    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:18.906414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.900026    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shv8p\" (UniqueName: \"kubernetes.io/projected/844ce432-ab27-4dbc-9ea2-10f933b38198-kube-api-access-shv8p\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:18.906414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.900045    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ce432-ab27-4dbc-9ea2-10f933b38198-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:18.945701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.945645    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:18.965412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:18.946018    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:19.536548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:19.536505    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq" event={"ID":"844ce432-ab27-4dbc-9ea2-10f933b38198","Type":"ContainerDied","Data":"3ae8687b0ed151b0208bc1df9c7bef1e12f1246ef4050f95ce546c9f15f97556"}
Nov 03 16:22:19.537101 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:19.537076    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae8687b0ed151b0208bc1df9c7bef1e12f1246ef4050f95ce546c9f15f97556"
Nov 03 16:22:19.548787 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:19.548760    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"
Nov 03 16:22:19.590281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:19.590194    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"]
Nov 03 16:22:19.892308 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:19.892207    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"]
Nov 03 16:22:20.085334 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.080520    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-afb990-mc8fn"]
Nov 03 16:22:20.187780 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:20.187721    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ec23d6_4838_498b_bbfc_c95776238998.slice/crio-conmon-ed27238aeedbf09944179c635daef09396382b1a8a717100fafd7d1343dc37bb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0538eea0_4223_42cf_88ac_648f96b8fe8e.slice/crio-conmon-a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb02ee0_e67d_4d6b_bfb7_f19fd3b2b1d1.slice/crio-conmon-3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb02ee0_e67d_4d6b_bfb7_f19fd3b2b1d1.slice/crio-3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0538eea0_4223_42cf_88ac_648f96b8fe8e.slice/crio-a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:20.603799 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.592983    4964 generic.go:356] "Generic (PLEG): container finished" podID="12ec23d6-4838-498b-bbfc-c95776238998" containerID="ed27238aeedbf09944179c635daef09396382b1a8a717100fafd7d1343dc37bb" exitCode=0
Nov 03 16:22:20.604207 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.603792    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" event={"ID":"12ec23d6-4838-498b-bbfc-c95776238998","Type":"ContainerDied","Data":"ed27238aeedbf09944179c635daef09396382b1a8a717100fafd7d1343dc37bb"}
Nov 03 16:22:20.622602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.619484    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:20.622602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.619544    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:20.622602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.621424    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:20.622602 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.621471    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:20.684555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.674788    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:22:20.684555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.674849    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:22:20.889793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.889757    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d754ffb0-1a73-4596-b18f-d7fdbbcccca6" path="/var/lib/kubelet/pods/d754ffb0-1a73-4596-b18f-d7fdbbcccca6/volumes"
Nov 03 16:22:20.908542 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:20.908507    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7" event={"ID":"e4299640-5495-4501-b8b7-25134ac691a4","Type":"ContainerStarted","Data":"e9e47a1b8b85b79329055780edd099c36aaf943d9564193facfe69352b465658"}
Nov 03 16:22:21.032748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.031048    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:22:21.145907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.134817    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:22:21.431817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.431786    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:22:21.448329 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448291    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448354    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448449    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448542    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448608    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448654    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448738    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448797    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.448891    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs\") pod \"12ec23d6-4838-498b-bbfc-c95776238998\" (UID: \"12ec23d6-4838-498b-bbfc-c95776238998\") "
Nov 03 16:22:21.449492 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.449290    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.449750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.449731    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.449971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.449953    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.452557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.450133    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.452557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.451258    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.452557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.451769    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:21.499548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.495014    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.499548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.495278    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "kube-api-access-sdzns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.499548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.496102    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.499548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.498893    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo" pod "12ec23d6-4838-498b-bbfc-c95776238998" (UID: "12ec23d6-4838-498b-bbfc-c95776238998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:21.550448 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550421    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ec23d6-4838-498b-bbfc-c95776238998-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550449    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzns\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-kube-api-access-sdzns\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550464    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550477    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550490    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550503    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ec23d6-4838-498b-bbfc-c95776238998-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550516    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ec23d6-4838-498b-bbfc-c95776238998-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.550561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.550529    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ec23d6-4838-498b-bbfc-c95776238998-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.568379 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.568349    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:22:21.570959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.570938    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:22:21.571751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.571710    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main" containerID="cri-o://1106fe7f3e3c80981cd955440b87cb32806a0c316e40e38485b11d27e20ddc29" gracePeriod=45
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651098    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651161    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651203    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651620    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651687    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651774    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651815    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651874    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651922    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651953    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.651989    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652093    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652124    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652164    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652192    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652222    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert\") pod \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\" (UID: \"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652255    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.652455 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.652323    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data\") pod \"0538eea0-4223-42cf-88ac-648f96b8fe8e\" (UID: \"0538eea0-4223-42cf-88ac-648f96b8fe8e\") "
Nov 03 16:22:21.653805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.653325    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.654043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654016    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:21.654135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654051    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "kube-api-access-m9jmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.654304 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654281    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.654366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654335    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"]
Nov 03 16:22:21.654530 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654504    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.654841 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654817    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.654919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.654852    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:21.655667 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.655641    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:21.655937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.655918    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.656074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.656060    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.656150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.656090    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.656207 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.656181    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.657043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.656987    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.657815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.657789    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:21.658100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.658075    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.659225 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.659189    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.659698 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.659670    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token" pod "0538eea0-4223-42cf-88ac-648f96b8fe8e" (UID: "0538eea0-4223-42cf-88ac-648f96b8fe8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.665939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.660709    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4" pod "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" (UID: "3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1"). InnerVolumeSpecName "kube-api-access-257q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758692    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257q4\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-kube-api-access-257q4\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758735    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758751    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0538eea0-4223-42cf-88ac-648f96b8fe8e-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758763    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758776    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758788    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758800    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758821    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758833    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758845    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758857    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758869    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758882    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758894    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758906    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9jmz\" (UniqueName: \"kubernetes.io/projected/0538eea0-4223-42cf-88ac-648f96b8fe8e-kube-api-access-m9jmz\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758918    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758930    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.759924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.758941    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0538eea0-4223-42cf-88ac-648f96b8fe8e-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.921445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.921368    4964 generic.go:356] "Generic (PLEG): container finished" podID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerID="a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842" exitCode=0
Nov 03 16:22:21.921730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.921485    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"
Nov 03 16:22:21.921730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.921629    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" event={"ID":"0538eea0-4223-42cf-88ac-648f96b8fe8e","Type":"ContainerDied","Data":"a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842"}
Nov 03 16:22:21.921730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.921686    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp" event={"ID":"0538eea0-4223-42cf-88ac-648f96b8fe8e","Type":"ContainerDied","Data":"be440bcf787697fad3b9f102887622bd1f4b035a22fb6dc010370a1b73944b91"}
Nov 03 16:22:21.921730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.921721    4964 scope.go:122] "RemoveContainer" containerID="a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842"
Nov 03 16:22:21.925182 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.925159    4964 generic.go:356] "Generic (PLEG): container finished" podID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerID="1106fe7f3e3c80981cd955440b87cb32806a0c316e40e38485b11d27e20ddc29" exitCode=0
Nov 03 16:22:21.925311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.925226    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" event={"ID":"5dacc118-e0f5-4abb-8caa-9547cb242155","Type":"ContainerDied","Data":"1106fe7f3e3c80981cd955440b87cb32806a0c316e40e38485b11d27e20ddc29"}
Nov 03 16:22:21.925311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.925278    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9" event={"ID":"5dacc118-e0f5-4abb-8caa-9547cb242155","Type":"ContainerDied","Data":"e13cc67a395c89c55414a7968b18cca9301f3228229e2f604a29ac8872c3eb4e"}
Nov 03 16:22:21.925311 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.925297    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e13cc67a395c89c55414a7968b18cca9301f3228229e2f604a29ac8872c3eb4e"
Nov 03 16:22:21.927536 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.927507    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx" event={"ID":"12ec23d6-4838-498b-bbfc-c95776238998","Type":"ContainerDied","Data":"c3a4d284516e349573f76c50ad93dcadecf1cc89ba430428bf79380261022599"}
Nov 03 16:22:21.927636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.927519    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"
Nov 03 16:22:21.930077 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.929826    4964 generic.go:356] "Generic (PLEG): container finished" podID="e4299640-5495-4501-b8b7-25134ac691a4" containerID="abace5763728e49fa9c4a57a6fec57cc57aa8498d96bf5e970e2d2a230c8b766" exitCode=0
Nov 03 16:22:21.930077 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.929912    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7" event={"ID":"e4299640-5495-4501-b8b7-25134ac691a4","Type":"ContainerDied","Data":"abace5763728e49fa9c4a57a6fec57cc57aa8498d96bf5e970e2d2a230c8b766"}
Nov 03 16:22:21.932774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.932751    4964 generic.go:356] "Generic (PLEG): container finished" podID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerID="3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586" exitCode=0
Nov 03 16:22:21.932894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.932790    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" event={"ID":"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1","Type":"ContainerDied","Data":"3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586"}
Nov 03 16:22:21.932894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.932811    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"
Nov 03 16:22:21.932894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.932830    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km" event={"ID":"3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1","Type":"ContainerDied","Data":"1ed6126faf9f6e06b71df311027ef174413e2246dd2234aa63d8d7b325dea77d"}
Nov 03 16:22:21.935933 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.935909    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:22:21.938624 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.938535    4964 scope.go:122] "RemoveContainer" containerID="a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842"
Nov 03 16:22:21.938907 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:21.938880    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842\": container with ID starting with a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842 not found: ID does not exist" containerID="a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842"
Nov 03 16:22:21.939017 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.938916    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842"} err="failed to get container status \"a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842\": rpc error: code = NotFound desc = could not find container \"a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842\": container with ID starting with a440819943b6ae96daf2c5f5f6d107316315749ed7c59008c9216d4375453842 not found: ID does not exist"
Nov 03 16:22:21.939017 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.938939    4964 scope.go:122] "RemoveContainer" containerID="ed27238aeedbf09944179c635daef09396382b1a8a717100fafd7d1343dc37bb"
Nov 03 16:22:21.951257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.951237    4964 scope.go:122] "RemoveContainer" containerID="3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586"
Nov 03 16:22:21.960189 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960159    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm\") pod \"5dacc118-e0f5-4abb-8caa-9547cb242155\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") "
Nov 03 16:22:21.960314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960216    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs\" (UniqueName: \"kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs\") pod \"5dacc118-e0f5-4abb-8caa-9547cb242155\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") "
Nov 03 16:22:21.960314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960286    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home\") pod \"5dacc118-e0f5-4abb-8caa-9547cb242155\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") "
Nov 03 16:22:21.960430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960324    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk\" (UniqueName: \"kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk\") pod \"5dacc118-e0f5-4abb-8caa-9547cb242155\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") "
Nov 03 16:22:21.960430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960371    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache\") pod \"5dacc118-e0f5-4abb-8caa-9547cb242155\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") "
Nov 03 16:22:21.960587 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960567    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir\") pod \"5dacc118-e0f5-4abb-8caa-9547cb242155\" (UID: \"5dacc118-e0f5-4abb-8caa-9547cb242155\") "
Nov 03 16:22:21.960793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.960645    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home" pod "5dacc118-e0f5-4abb-8caa-9547cb242155" (UID: "5dacc118-e0f5-4abb-8caa-9547cb242155"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.961231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.961208    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache" pod "5dacc118-e0f5-4abb-8caa-9547cb242155" (UID: "5dacc118-e0f5-4abb-8caa-9547cb242155"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.961340 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.961323    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.961430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.961339    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:21.961510 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.961449    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir" pod "5dacc118-e0f5-4abb-8caa-9547cb242155" (UID: "5dacc118-e0f5-4abb-8caa-9547cb242155"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.963722 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.963697    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm" pod "5dacc118-e0f5-4abb-8caa-9547cb242155" (UID: "5dacc118-e0f5-4abb-8caa-9547cb242155"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:21.963805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.963736    4964 scope.go:122] "RemoveContainer" containerID="3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586"
Nov 03 16:22:21.964096 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:21.964043    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586\": container with ID starting with 3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586 not found: ID does not exist" containerID="3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586"
Nov 03 16:22:21.964096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.964083    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586"} err="failed to get container status \"3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586\": rpc error: code = NotFound desc = could not find container \"3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586\": container with ID starting with 3f72c849ac227fa4c39e6e63ea23f5ab9cb435acea9ba0bd5e778787aeeca586 not found: ID does not exist"
Nov 03 16:22:21.964427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.964210    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs" pod "5dacc118-e0f5-4abb-8caa-9547cb242155" (UID: "5dacc118-e0f5-4abb-8caa-9547cb242155"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:21.964518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.964470    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk" pod "5dacc118-e0f5-4abb-8caa-9547cb242155" (UID: "5dacc118-e0f5-4abb-8caa-9547cb242155"). InnerVolumeSpecName "kube-api-access-q86jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:21.981611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.981585    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"]
Nov 03 16:22:21.994561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:21.994535    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-zfkvx"]
Nov 03 16:22:22.020147 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.020105    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"]
Nov 03 16:22:22.027429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.027365    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-gvzkp"]
Nov 03 16:22:22.052257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.052139    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"]
Nov 03 16:22:22.059358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.059321    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-qg6km"]
Nov 03 16:22:22.062639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.062613    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.062639 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.062639    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dacc118-e0f5-4abb-8caa-9547cb242155-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.062804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.062650    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q86jk\" (UniqueName: \"kubernetes.io/projected/5dacc118-e0f5-4abb-8caa-9547cb242155-kube-api-access-q86jk\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.062804 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.062661    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dacc118-e0f5-4abb-8caa-9547cb242155-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:22.609117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.609074    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" path="/var/lib/kubelet/pods/0538eea0-4223-42cf-88ac-648f96b8fe8e/volumes"
Nov 03 16:22:22.609683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.609660    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ec23d6-4838-498b-bbfc-c95776238998" path="/var/lib/kubelet/pods/12ec23d6-4838-498b-bbfc-c95776238998/volumes"
Nov 03 16:22:22.610162 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.610143    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" path="/var/lib/kubelet/pods/3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1/volumes"
Nov 03 16:22:22.944334 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:22.944306    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"
Nov 03 16:22:23.024169 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.024133    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"]
Nov 03 16:22:23.031821 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.031788    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/simulator-w4-4a19cd-kserve-74b5fb998f-zpjs9"]
Nov 03 16:22:23.132450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.132418    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:23.179237 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.179203    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca\") pod \"e4299640-5495-4501-b8b7-25134ac691a4\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") "
Nov 03 16:22:23.179408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.179254    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca\") pod \"e4299640-5495-4501-b8b7-25134ac691a4\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") "
Nov 03 16:22:23.179408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.179343    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb\" (UniqueName: \"kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb\") pod \"e4299640-5495-4501-b8b7-25134ac691a4\" (UID: \"e4299640-5495-4501-b8b7-25134ac691a4\") "
Nov 03 16:22:23.179734 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.179711    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca" pod "e4299640-5495-4501-b8b7-25134ac691a4" (UID: "e4299640-5495-4501-b8b7-25134ac691a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:23.181722 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.181685    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca" pod "e4299640-5495-4501-b8b7-25134ac691a4" (UID: "e4299640-5495-4501-b8b7-25134ac691a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:23.181722 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.181708    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb" pod "e4299640-5495-4501-b8b7-25134ac691a4" (UID: "e4299640-5495-4501-b8b7-25134ac691a4"). InnerVolumeSpecName "kube-api-access-4dnhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:23.280927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.280840    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:23.280927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.280878    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4299640-5495-4501-b8b7-25134ac691a4-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:23.280927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.280892    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnhb\" (UniqueName: \"kubernetes.io/projected/e4299640-5495-4501-b8b7-25134ac691a4-kube-api-access-4dnhb\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:23.954470 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.954438    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"
Nov 03 16:22:23.954971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.954446    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7" event={"ID":"e4299640-5495-4501-b8b7-25134ac691a4","Type":"ContainerDied","Data":"e9e47a1b8b85b79329055780edd099c36aaf943d9564193facfe69352b465658"}
Nov 03 16:22:23.954971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:23.954576    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e47a1b8b85b79329055780edd099c36aaf943d9564193facfe69352b465658"
Nov 03 16:22:24.613011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:24.612965    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" path="/var/lib/kubelet/pods/5dacc118-e0f5-4abb-8caa-9547cb242155/volumes"
Nov 03 16:22:25.235005 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:25.234964    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:25.305194 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.305158    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"]
Nov 03 16:22:25.305733 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.305698    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" containerID="cri-o://fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a" gracePeriod=30
Nov 03 16:22:25.311817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.311789    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"]
Nov 03 16:22:25.312042 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.311997    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy" containerID="cri-o://a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947" gracePeriod=30
Nov 03 16:22:25.318096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.318064    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"]
Nov 03 16:22:25.318342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.318299    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" containerID="cri-o://13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de" gracePeriod=30
Nov 03 16:22:25.499948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.499257    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:25.499948 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.499334    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:25.503317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.503277    4964 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:25.503626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:25.503599    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:28.148425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148365    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"]
Nov 03 16:22:28.148937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148922    4964 cpu_manager.go:414] "containerMap: removing container" podUID="844ce432-ab27-4dbc-9ea2-10f933b38198" containerName="revoke-keys"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148940    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844ce432-ab27-4dbc-9ea2-10f933b38198" containerName="revoke-keys"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148956    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148964    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148973    4964 cpu_manager.go:414] "containerMap: removing container" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148978    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148994    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e4299640-5495-4501-b8b7-25134ac691a4" containerName="revoke-keys"
Nov 03 16:22:28.149001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.148999    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4299640-5495-4501-b8b7-25134ac691a4" containerName="revoke-keys"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149007    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149011    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149020    4964 cpu_manager.go:414] "containerMap: removing container" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149024    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149109    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="844ce432-ab27-4dbc-9ea2-10f933b38198" containerName="revoke-keys"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149116    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb02ee0-e67d-4d6b-bfb7-f19fd3b2b1d1" containerName="istio-proxy"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149122    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ec23d6-4838-498b-bbfc-c95776238998" containerName="istio-proxy"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149130    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4299640-5495-4501-b8b7-25134ac691a4" containerName="revoke-keys"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149137    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dacc118-e0f5-4abb-8caa-9547cb242155" containerName="main"
Nov 03 16:22:28.149231 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.149144    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="0538eea0-4223-42cf-88ac-648f96b8fe8e" containerName="istio-proxy"
Nov 03 16:22:28.153998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.153973    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.159362 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.159339    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:22:28.159494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.159378    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:22:28.169205 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.169162    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"]
Nov 03 16:22:28.255326 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.255286    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.255517 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.255359    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.255590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.255559    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxv7\" (UniqueName: \"kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.257611 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.257594    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:22:28.257688 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.257671    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:22:28.357261 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.357227    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxv7\" (UniqueName: \"kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.357458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.357338    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.357529 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.357459    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.358201 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.358177    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.359832 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.359815    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.380371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.380338    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxv7\" (UniqueName: \"kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7\") pod \"maas-api-revoke-sub-ed08bf69-l4tz9\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.468660 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.468621    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:28.800043 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.799998    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"]
Nov 03 16:22:28.815367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.815320    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"]
Nov 03 16:22:28.815651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.815592    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy" containerID="cri-o://446c4e69c52c2a76d7ba95e6211136008e2aca9396c9075151a748cd24d2f93a" gracePeriod=30
Nov 03 16:22:28.820218 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.820189    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"]
Nov 03 16:22:28.820483 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.820440    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy" containerID="cri-o://030c7e0791831af202fa956dbeb4fa6fad61e8cfdb4ff9fb760d400fb8b27728" gracePeriod=30
Nov 03 16:22:28.841556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:28.841515    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"]
Nov 03 16:22:29.025704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:29.025540    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9" event={"ID":"f8882c3a-731c-4618-9243-3b1eed5606fa","Type":"ContainerStarted","Data":"d5aa095f21e02e1b1f2da9081860f5b9cef1c114ffa04ba0463b5548d63bdead"}
Nov 03 16:22:29.025704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:29.025656    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" containerID="cri-o://5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d" gracePeriod=30
Nov 03 16:22:29.531767 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:29.531722    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"]
Nov 03 16:22:29.536409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:29.536370    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4a19cd-2wzq4"]
Nov 03 16:22:30.035298 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.035261    4964 generic.go:356] "Generic (PLEG): container finished" podID="f8882c3a-731c-4618-9243-3b1eed5606fa" containerID="b7ec7b8fc63e012227e7dafcfe409f0567c811d9eec8c5b88fa4f9af16cbf094" exitCode=0
Nov 03 16:22:30.035490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.035319    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9" event={"ID":"f8882c3a-731c-4618-9243-3b1eed5606fa","Type":"ContainerDied","Data":"b7ec7b8fc63e012227e7dafcfe409f0567c811d9eec8c5b88fa4f9af16cbf094"}
Nov 03 16:22:30.612419 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.612375    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64288cf0-1310-4f63-aa0f-cc124c8b3bff" path="/var/lib/kubelet/pods/64288cf0-1310-4f63-aa0f-cc124c8b3bff/volumes"
Nov 03 16:22:30.619303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.619279    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:22:30.671314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.671285    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:22:30.679314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.679283    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:22:30.698509 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698469    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698596    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698657    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698688    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698750    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698780    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698798    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.698840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698808    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.698840 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698834    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.699042 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.698861    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo\") pod \"6a800b84-3440-4c9a-9084-5ff793a7082a\" (UID: \"6a800b84-3440-4c9a-9084-5ff793a7082a\") "
Nov 03 16:22:30.699148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.699134    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.699185 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.699142    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.699635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.699439    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.699635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.699439    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:30.699635 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.699599    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.701670 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.701635    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:30.702031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.701988    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.702137 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.702090    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.702431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.702372    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd" pod "6a800b84-3440-4c9a-9084-5ff793a7082a" (UID: "6a800b84-3440-4c9a-9084-5ff793a7082a"). InnerVolumeSpecName "kube-api-access-x95wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.799996 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.799952    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800195 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800091    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800195 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800151    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800193    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800233    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800260    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800302    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800338    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800366    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800429    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800437    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800471    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800504    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800586 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800535    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800593    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800645    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800676    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket\") pod \"242af026-10c3-4aba-baec-ef113d93d527\" (UID: \"242af026-10c3-4aba-baec-ef113d93d527\") "
Nov 03 16:22:30.800923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800720    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.800923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800746    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.800923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800756    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket\") pod \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\" (UID: \"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2\") "
Nov 03 16:22:30.801202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.800957    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.801202 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.801164    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.801408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.801364    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.801560 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.801171    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.801705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.801644    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.802307 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.802285    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:30.802690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.802670    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806280    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806651    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.801456    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806700    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806719    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806732    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806745    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242af026-10c3-4aba-baec-ef113d93d527-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806759    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806772    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806786    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a800b84-3440-4c9a-9084-5ff793a7082a-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806802    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95wd\" (UniqueName: \"kubernetes.io/projected/6a800b84-3440-4c9a-9084-5ff793a7082a-kube-api-access-x95wd\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806818    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a800b84-3440-4c9a-9084-5ff793a7082a-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806831    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.807210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.806845    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.807712    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.807881    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.807892    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "kube-api-access-hsfkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.808095    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token" pod "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" (UID: "9e9c1550-4b46-4a0f-b6e7-771f8691dcf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.808680    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.808712    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "kube-api-access-lwb8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.808800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.808727    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token" pod "242af026-10c3-4aba-baec-ef113d93d527" (UID: "242af026-10c3-4aba-baec-ef113d93d527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:30.907919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907883    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907929    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907944    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907958    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907971    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsfkx\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-kube-api-access-hsfkx\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907984    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.907996    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwb8c\" (UniqueName: \"kubernetes.io/projected/242af026-10c3-4aba-baec-ef113d93d527-kube-api-access-lwb8c\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908010    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908023    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908035    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908047    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242af026-10c3-4aba-baec-ef113d93d527-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908059    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908072    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:30.908128 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:30.908085    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242af026-10c3-4aba-baec-ef113d93d527-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:31.043143 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.043106    4964 generic.go:356] "Generic (PLEG): container finished" podID="242af026-10c3-4aba-baec-ef113d93d527" containerID="13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de" exitCode=0
Nov 03 16:22:31.043350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.043163    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"
Nov 03 16:22:31.043350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.043187    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" event={"ID":"242af026-10c3-4aba-baec-ef113d93d527","Type":"ContainerDied","Data":"13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de"}
Nov 03 16:22:31.043350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.043238    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz" event={"ID":"242af026-10c3-4aba-baec-ef113d93d527","Type":"ContainerDied","Data":"5f518b8e97777fb636991c734bc6265bdc6efea57e0d448a6303509b815d7dee"}
Nov 03 16:22:31.043350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.043265    4964 scope.go:122] "RemoveContainer" containerID="13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de"
Nov 03 16:22:31.044824 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.044798    4964 generic.go:356] "Generic (PLEG): container finished" podID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerID="fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a" exitCode=0
Nov 03 16:22:31.044991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.044854    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"
Nov 03 16:22:31.044991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.044880    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" event={"ID":"6a800b84-3440-4c9a-9084-5ff793a7082a","Type":"ContainerDied","Data":"fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a"}
Nov 03 16:22:31.044991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.044922    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv" event={"ID":"6a800b84-3440-4c9a-9084-5ff793a7082a","Type":"ContainerDied","Data":"a4e8768ab8b7bac721e6fb13ef1d32b181b908e554d28feb27e3dd92b1eda309"}
Nov 03 16:22:31.046337 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.046315    4964 generic.go:356] "Generic (PLEG): container finished" podID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerID="a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947" exitCode=0
Nov 03 16:22:31.046458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.046362    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"
Nov 03 16:22:31.046458 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.046428    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" event={"ID":"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2","Type":"ContainerDied","Data":"a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947"}
Nov 03 16:22:31.046590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.046481    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl" event={"ID":"9e9c1550-4b46-4a0f-b6e7-771f8691dcf2","Type":"ContainerDied","Data":"9371023e31712362d0499b949b52b4c588b27d8b8fae12ee90c11bcf0f34a26f"}
Nov 03 16:22:31.057380 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.057360    4964 scope.go:122] "RemoveContainer" containerID="13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de"
Nov 03 16:22:31.057696 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:31.057672    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de\": container with ID starting with 13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de not found: ID does not exist" containerID="13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de"
Nov 03 16:22:31.057793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.057704    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de"} err="failed to get container status \"13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de\": rpc error: code = NotFound desc = could not find container \"13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de\": container with ID starting with 13741527e33d52a3ea507a02b1db2719c92fa8be190b3899615aef61d16a48de not found: ID does not exist"
Nov 03 16:22:31.057793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.057726    4964 scope.go:122] "RemoveContainer" containerID="fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a"
Nov 03 16:22:31.070705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.070322    4964 scope.go:122] "RemoveContainer" containerID="fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a"
Nov 03 16:22:31.070835 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:31.070713    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a\": container with ID starting with fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a not found: ID does not exist" containerID="fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a"
Nov 03 16:22:31.070835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.070741    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a"} err="failed to get container status \"fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a\": rpc error: code = NotFound desc = could not find container \"fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a\": container with ID starting with fbd0625f1b7591109586ff6ecc920ca8d590ea54b5dc937bc4a6d3f1f763811a not found: ID does not exist"
Nov 03 16:22:31.070835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.070765    4964 scope.go:122] "RemoveContainer" containerID="a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947"
Nov 03 16:22:31.085183 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.085062    4964 scope.go:122] "RemoveContainer" containerID="a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947"
Nov 03 16:22:31.085501 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:31.085471    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947\": container with ID starting with a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947 not found: ID does not exist" containerID="a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947"
Nov 03 16:22:31.085583 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.085515    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947"} err="failed to get container status \"a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947\": rpc error: code = NotFound desc = could not find container \"a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947\": container with ID starting with a6afaaf576cb1cceb9bd0af2a50f278ec0112d8b80cd424984f905c17548e947 not found: ID does not exist"
Nov 03 16:22:31.119636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.119600    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"]
Nov 03 16:22:31.124816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.124784    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-nmstv"]
Nov 03 16:22:31.134449 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.134417    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"]
Nov 03 16:22:31.140004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.139963    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-csdzl"]
Nov 03 16:22:31.158997 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.158919    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"]
Nov 03 16:22:31.164716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.164687    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-vj9cz"]
Nov 03 16:22:31.213876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.213842    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:31.312817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.312766    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca\") pod \"f8882c3a-731c-4618-9243-3b1eed5606fa\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") "
Nov 03 16:22:31.312817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.312828    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7\" (UniqueName: \"kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7\") pod \"f8882c3a-731c-4618-9243-3b1eed5606fa\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") "
Nov 03 16:22:31.313038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.312997    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca\") pod \"f8882c3a-731c-4618-9243-3b1eed5606fa\" (UID: \"f8882c3a-731c-4618-9243-3b1eed5606fa\") "
Nov 03 16:22:31.313265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.313234    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca" pod "f8882c3a-731c-4618-9243-3b1eed5606fa" (UID: "f8882c3a-731c-4618-9243-3b1eed5606fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:31.313431 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.313381    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:31.315115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.315084    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca" pod "f8882c3a-731c-4618-9243-3b1eed5606fa" (UID: "f8882c3a-731c-4618-9243-3b1eed5606fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:31.315254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.315114    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7" pod "f8882c3a-731c-4618-9243-3b1eed5606fa" (UID: "f8882c3a-731c-4618-9243-3b1eed5606fa"). InnerVolumeSpecName "kube-api-access-xsxv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:31.414704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.414613    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8882c3a-731c-4618-9243-3b1eed5606fa-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:31.414704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:31.414645    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxv7\" (UniqueName: \"kubernetes.io/projected/f8882c3a-731c-4618-9243-3b1eed5606fa-kube-api-access-xsxv7\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:32.055696 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.055665    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"
Nov 03 16:22:32.056149 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.055694    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9" event={"ID":"f8882c3a-731c-4618-9243-3b1eed5606fa","Type":"ContainerDied","Data":"d5aa095f21e02e1b1f2da9081860f5b9cef1c114ffa04ba0463b5548d63bdead"}
Nov 03 16:22:32.056149 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.055747    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5aa095f21e02e1b1f2da9081860f5b9cef1c114ffa04ba0463b5548d63bdead"
Nov 03 16:22:32.567532 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:32.567490    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:22:32.609723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.609690    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242af026-10c3-4aba-baec-ef113d93d527" path="/var/lib/kubelet/pods/242af026-10c3-4aba-baec-ef113d93d527/volumes"
Nov 03 16:22:32.610176 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.610163    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" path="/var/lib/kubelet/pods/6a800b84-3440-4c9a-9084-5ff793a7082a/volumes"
Nov 03 16:22:32.610579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.610567    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" path="/var/lib/kubelet/pods/9e9c1550-4b46-4a0f-b6e7-771f8691dcf2/volumes"
Nov 03 16:22:32.709505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.709470    4964 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:32.709653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:32.709531    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:34.079929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.079890    4964 generic.go:356] "Generic (PLEG): container finished" podID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerID="030c7e0791831af202fa956dbeb4fa6fad61e8cfdb4ff9fb760d400fb8b27728" exitCode=0
Nov 03 16:22:34.080523 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.080135    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" event={"ID":"1fe45f54-0726-4d25-a8c5-1f582fa55534","Type":"ContainerDied","Data":"030c7e0791831af202fa956dbeb4fa6fad61e8cfdb4ff9fb760d400fb8b27728"}
Nov 03 16:22:34.083060 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.083032    4964 generic.go:356] "Generic (PLEG): container finished" podID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerID="446c4e69c52c2a76d7ba95e6211136008e2aca9396c9075151a748cd24d2f93a" exitCode=0
Nov 03 16:22:34.083210 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.083159    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" event={"ID":"774f3575-ea70-41a8-92bd-df711b8eb36c","Type":"ContainerDied","Data":"446c4e69c52c2a76d7ba95e6211136008e2aca9396c9075151a748cd24d2f93a"}
Nov 03 16:22:34.154749 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.154718    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:22:34.159760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.158829    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246152    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246201    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246313    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246342    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246378    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246424    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246488    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246531    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246592    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246660    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246701    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246751    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246794    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246825    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246862    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246899    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket\") pod \"774f3575-ea70-41a8-92bd-df711b8eb36c\" (UID: \"774f3575-ea70-41a8-92bd-df711b8eb36c\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.246933    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data\") pod \"1fe45f54-0726-4d25-a8c5-1f582fa55534\" (UID: \"1fe45f54-0726-4d25-a8c5-1f582fa55534\") "
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.247684    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.248442 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.248101    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.249626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.248874    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.249626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.249052    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.249626 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.249222    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.249799 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.249640    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.249928 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.249906    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.250711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.250561    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.251109 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.250852    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.252191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.252094    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.254864 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.254786    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "kube-api-access-pngmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.256620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.256570    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.256620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.256593    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.256975 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.256933    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.258622 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.257548    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.258622 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.257604    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy" pod "1fe45f54-0726-4d25-a8c5-1f582fa55534" (UID: "1fe45f54-0726-4d25-a8c5-1f582fa55534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.258622 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.258587    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.258622 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.258602    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw" pod "774f3575-ea70-41a8-92bd-df711b8eb36c" (UID: "774f3575-ea70-41a8-92bd-df711b8eb36c"). InnerVolumeSpecName "kube-api-access-ggpbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.333681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.333651    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:22:34.348793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348749    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.348793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348780    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.348793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348790    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.348793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348802    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348812    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngmd\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-kube-api-access-pngmd\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348821    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348831    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348840    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348863    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774f3575-ea70-41a8-92bd-df711b8eb36c-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348873    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348882    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348891    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348899    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348909    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpbw\" (UniqueName: \"kubernetes.io/projected/774f3575-ea70-41a8-92bd-df711b8eb36c-kube-api-access-ggpbw\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348918    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348925    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe45f54-0726-4d25-a8c5-1f582fa55534-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348933    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe45f54-0726-4d25-a8c5-1f582fa55534-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.349140 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.348941    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774f3575-ea70-41a8-92bd-df711b8eb36c-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.449365 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449325    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449377    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449443    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449520    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449552    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449599    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449650    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.449810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449660    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.449810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449682    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.450009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449816    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token\") pod \"f6b920e4-9c84-4f35-981e-1a95da93e099\" (UID: \"f6b920e4-9c84-4f35-981e-1a95da93e099\") "
Nov 03 16:22:34.450009 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.449891    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.450308 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.450219    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.450308 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.450245    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.450518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.450418    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:34.450518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.450438    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.450518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.450509    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.452378 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.452353    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:34.452526 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.452408    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "kube-api-access-4k59w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.453066 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.453003    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:34.453172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.453090    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo" pod "f6b920e4-9c84-4f35-981e-1a95da93e099" (UID: "f6b920e4-9c84-4f35-981e-1a95da93e099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:34.551374 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551331    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k59w\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-kube-api-access-4k59w\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.551374 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551367    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.551374 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551381    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.551651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551408    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b920e4-9c84-4f35-981e-1a95da93e099-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.551651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551421    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.551651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551432    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b920e4-9c84-4f35-981e-1a95da93e099-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.551651 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:34.551443    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b920e4-9c84-4f35-981e-1a95da93e099-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:34.837287 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:34.837188    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:35.091901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.091816    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"
Nov 03 16:22:35.091901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.091816    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt" event={"ID":"774f3575-ea70-41a8-92bd-df711b8eb36c","Type":"ContainerDied","Data":"27714edadd17d5719130084c5e596581d32cc8d55ff37f502afd3c66cba3bffa"}
Nov 03 16:22:35.092459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.091983    4964 scope.go:122] "RemoveContainer" containerID="446c4e69c52c2a76d7ba95e6211136008e2aca9396c9075151a748cd24d2f93a"
Nov 03 16:22:35.093369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.093306    4964 generic.go:356] "Generic (PLEG): container finished" podID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerID="5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d" exitCode=0
Nov 03 16:22:35.093457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.093414    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"
Nov 03 16:22:35.093457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.093420    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" event={"ID":"f6b920e4-9c84-4f35-981e-1a95da93e099","Type":"ContainerDied","Data":"5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d"}
Nov 03 16:22:35.093571 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.093467    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf" event={"ID":"f6b920e4-9c84-4f35-981e-1a95da93e099","Type":"ContainerDied","Data":"515ad46a5f4758c7e0071df46e867294837073af5fddba3b15de94f3dcead09e"}
Nov 03 16:22:35.095227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.095135    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"
Nov 03 16:22:35.095227 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.095169    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm" event={"ID":"1fe45f54-0726-4d25-a8c5-1f582fa55534","Type":"ContainerDied","Data":"34490c632312622f245bf341248f4d498468ff5b7f3578172fb271cf6f0225af"}
Nov 03 16:22:35.103500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.103480    4964 scope.go:122] "RemoveContainer" containerID="5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d"
Nov 03 16:22:35.113720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.113696    4964 scope.go:122] "RemoveContainer" containerID="5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d"
Nov 03 16:22:35.114103 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:35.114077    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d\": container with ID starting with 5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d not found: ID does not exist" containerID="5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d"
Nov 03 16:22:35.114167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.114114    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d"} err="failed to get container status \"5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d\": rpc error: code = NotFound desc = could not find container \"5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d\": container with ID starting with 5ae161b3c4f3642f347d1fac47771cf4873c82a12b5e0e7d9f80cc652a87321d not found: ID does not exist"
Nov 03 16:22:35.114167 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.114131    4964 scope.go:122] "RemoveContainer" containerID="030c7e0791831af202fa956dbeb4fa6fad61e8cfdb4ff9fb760d400fb8b27728"
Nov 03 16:22:35.125314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.125284    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"]
Nov 03 16:22:35.130181 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.130157    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-w7qqt"]
Nov 03 16:22:35.141291 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.141262    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"]
Nov 03 16:22:35.145027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.145002    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-rbvzm"]
Nov 03 16:22:35.160810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.160765    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"]
Nov 03 16:22:35.165578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:35.165550    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-kx5xf"]
Nov 03 16:22:35.252737 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:35.252704    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:36.608225 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:36.608189    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" path="/var/lib/kubelet/pods/1fe45f54-0726-4d25-a8c5-1f582fa55534/volumes"
Nov 03 16:22:36.609232 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:36.609206    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" path="/var/lib/kubelet/pods/774f3575-ea70-41a8-92bd-df711b8eb36c/volumes"
Nov 03 16:22:36.609753 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:36.609733    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" path="/var/lib/kubelet/pods/f6b920e4-9c84-4f35-981e-1a95da93e099/volumes"
Nov 03 16:22:36.691657 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:36.691605    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" containerID="cri-o://12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2" gracePeriod=45
Nov 03 16:22:36.813991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:36.813954    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": dial tcp 10.132.0.52:8000: connect: connection refused"
Nov 03 16:22:36.972972 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:36.972937    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.095822    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir\") pod \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") "
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.095873    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache\") pod \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") "
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.095971    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs\") pod \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") "
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.096013    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home\") pod \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") "
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.096069    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj\" (UniqueName: \"kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj\") pod \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") "
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.096168    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm\") pod \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\" (UID: \"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469\") "
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.097213    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir" pod "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" (UID: "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.097488    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache" pod "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" (UID: "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.097759 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.097643    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home" pod "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" (UID: "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.104469 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.101720    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs" pod "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" (UID: "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:22:37.104469 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.102561    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm" pod "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" (UID: "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:37.133478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.120955    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj" pod "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" (UID: "ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469"). InnerVolumeSpecName "kube-api-access-z9zmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:37.133478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.124820    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"]
Nov 03 16:22:37.133478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.125050    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" containerID="cri-o://1881b9d20f65cd33e2ab25024197afb2cedc5b77e11dfe777daeee311bbfb5fd" gracePeriod=30
Nov 03 16:22:37.135646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.135381    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"]
Nov 03 16:22:37.135787 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.135643    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy" containerID="cri-o://1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd" gracePeriod=30
Nov 03 16:22:37.142204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.142170    4964 generic.go:356] "Generic (PLEG): container finished" podID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerID="12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2" exitCode=0
Nov 03 16:22:37.142708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.142455    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" event={"ID":"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469","Type":"ContainerDied","Data":"12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2"}
Nov 03 16:22:37.142708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.142497    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc" event={"ID":"ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469","Type":"ContainerDied","Data":"0ec2803a2de64e9cef32b83947eb61fd27f7f264d7c5826640864b6f867a0787"}
Nov 03 16:22:37.142708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.142520    4964 scope.go:122] "RemoveContainer" containerID="12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2"
Nov 03 16:22:37.142708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.142546    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"
Nov 03 16:22:37.155810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.155774    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"]
Nov 03 16:22:37.156032 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.156009    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" containerID="cri-o://f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9" gracePeriod=30
Nov 03 16:22:37.165963 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.165933    4964 scope.go:122] "RemoveContainer" containerID="12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2"
Nov 03 16:22:37.166755 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:37.166722    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2\": container with ID starting with 12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2 not found: ID does not exist" containerID="12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2"
Nov 03 16:22:37.166875 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.166768    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2"} err="failed to get container status \"12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2\": rpc error: code = NotFound desc = could not find container \"12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2\": container with ID starting with 12556b687636a2544dc76289bb1b02373937dc18cd96507a5a3e22b10b920dd2 not found: ID does not exist"
Nov 03 16:22:37.198098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.198049    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tmp-dir\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.198098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.198091    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-model-cache\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.198098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.198106    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-tls-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.200959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.198122    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-home\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.200959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.198139    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9zmj\" (UniqueName: \"kubernetes.io/projected/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-kube-api-access-z9zmj\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.200959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.198152    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469-dshm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:37.272575 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.272530    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"]
Nov 03 16:22:37.276125 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:37.276082    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/simulator-w2-b132e3-kserve-675cb6fbcc-m2wtc"]
Nov 03 16:22:38.609197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:38.609161    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" path="/var/lib/kubelet/pods/ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469/volumes"
Nov 03 16:22:40.277932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:40.277880    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"]
Nov 03 16:22:40.300766 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:40.299543    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b132e3-jk5s7"]
Nov 03 16:22:40.611732 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:40.611652    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4299640-5495-4501-b8b7-25134ac691a4" path="/var/lib/kubelet/pods/e4299640-5495-4501-b8b7-25134ac691a4/volumes"
Nov 03 16:22:40.990068 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:40.990034    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:40.990235 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:40.990094    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:42.103899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.103861    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:22:42.235543 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.235465    4964 generic.go:356] "Generic (PLEG): container finished" podID="89b68376-e690-403a-a38b-52897ffaf73a" containerID="1881b9d20f65cd33e2ab25024197afb2cedc5b77e11dfe777daeee311bbfb5fd" exitCode=0
Nov 03 16:22:42.235749 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.235575    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" event={"ID":"89b68376-e690-403a-a38b-52897ffaf73a","Type":"ContainerDied","Data":"1881b9d20f65cd33e2ab25024197afb2cedc5b77e11dfe777daeee311bbfb5fd"}
Nov 03 16:22:42.480428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.479266    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:22:42.512942 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.512890    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:22:42.595615 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.595575    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:22:42.595811 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.595660    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:22:42.665910 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.665868    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.665910 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.665909    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.666148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.665927    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.665961    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.666148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666009    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.666148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666045    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666102    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.666148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666130    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.666505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666181    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666209    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666202    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.666505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666221    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.666505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666457    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.666505 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666469    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.666794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666588    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666674    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666680    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.666794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666704    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.666794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666732    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.666794 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666753    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.667271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666826    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs\") pod \"246d37ae-a202-4576-9c03-f041c43410df\" (UID: \"246d37ae-a202-4576-9c03-f041c43410df\") "
Nov 03 16:22:42.667271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666848    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.667271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666866    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo\") pod \"15070bd9-7432-4a73-80c0-f9cd4af75282\" (UID: \"15070bd9-7432-4a73-80c0-f9cd4af75282\") "
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667286    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667308    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667322    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667338    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667352    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.666864    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667038    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:42.667457 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.667060    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.668339 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.668311    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:42.669354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.669306    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.669821 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.669792    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.669923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.669861    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:42.670197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.670176    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.670197 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.670185    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "kube-api-access-fqc85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.670364 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.670205    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy" pod "15070bd9-7432-4a73-80c0-f9cd4af75282" (UID: "15070bd9-7432-4a73-80c0-f9cd4af75282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.670724 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.670702    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "kube-api-access-554t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.671495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.671469    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.671495 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.671480    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo" pod "246d37ae-a202-4576-9c03-f041c43410df" (UID: "246d37ae-a202-4576-9c03-f041c43410df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:42.768130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768092    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768128    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqc85\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-kube-api-access-fqc85\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768140    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768151    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768161    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768169    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768177    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768185    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d37ae-a202-4576-9c03-f041c43410df-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768193    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d37ae-a202-4576-9c03-f041c43410df-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768201    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d37ae-a202-4576-9c03-f041c43410df-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768209    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/246d37ae-a202-4576-9c03-f041c43410df-kube-api-access-554t5\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768217    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15070bd9-7432-4a73-80c0-f9cd4af75282-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.768655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.768225    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15070bd9-7432-4a73-80c0-f9cd4af75282-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.793579 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.793547    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:22:42.970254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970175    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970254 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970247    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970335    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970357    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970379    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970429    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970480    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970491    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.970868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970560    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket\") pod \"89b68376-e690-403a-a38b-52897ffaf73a\" (UID: \"89b68376-e690-403a-a38b-52897ffaf73a\") "
Nov 03 16:22:42.970868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970608    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.970868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970746    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.971027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970925    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:42.971027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970964    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.971027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970981    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.971027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.970990    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:42.971230 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.971075    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.972687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.972664    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:42.972687 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.972676    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:42.972817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.972705    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:42.972817 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:42.972711    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g" pod "89b68376-e690-403a-a38b-52897ffaf73a" (UID: "89b68376-e690-403a-a38b-52897ffaf73a"). InnerVolumeSpecName "kube-api-access-jqc5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:43.071807 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.071760    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqc5g\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-kube-api-access-jqc5g\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.071958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.071836    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b68376-e690-403a-a38b-52897ffaf73a-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.071958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.071853    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b68376-e690-403a-a38b-52897ffaf73a-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.071958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.071866    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b68376-e690-403a-a38b-52897ffaf73a-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.071958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.071879    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.071958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.071891    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b68376-e690-403a-a38b-52897ffaf73a-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:43.245211 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.245097    4964 generic.go:356] "Generic (PLEG): container finished" podID="246d37ae-a202-4576-9c03-f041c43410df" containerID="1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd" exitCode=0
Nov 03 16:22:43.245735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.245357    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" event={"ID":"246d37ae-a202-4576-9c03-f041c43410df","Type":"ContainerDied","Data":"1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd"}
Nov 03 16:22:43.245735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.245472    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb" event={"ID":"246d37ae-a202-4576-9c03-f041c43410df","Type":"ContainerDied","Data":"05e4b16e7be7bb453ca77cc4c4cd237763e6077e1389b571df3dde04ad27204b"}
Nov 03 16:22:43.245735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.245513    4964 scope.go:122] "RemoveContainer" containerID="1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd"
Nov 03 16:22:43.245735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.245690    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"
Nov 03 16:22:43.250289 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.250257    4964 generic.go:356] "Generic (PLEG): container finished" podID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerID="f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9" exitCode=0
Nov 03 16:22:43.250435 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.250332    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"
Nov 03 16:22:43.250435 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.250420    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" event={"ID":"15070bd9-7432-4a73-80c0-f9cd4af75282","Type":"ContainerDied","Data":"f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9"}
Nov 03 16:22:43.250606 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.250462    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998" event={"ID":"15070bd9-7432-4a73-80c0-f9cd4af75282","Type":"ContainerDied","Data":"97a2264f1f4fb743c69b2e76de4027401156c80fe6a82945b33b30663f922376"}
Nov 03 16:22:43.255257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.255230    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm" event={"ID":"89b68376-e690-403a-a38b-52897ffaf73a","Type":"ContainerDied","Data":"f2ce79f2c83faa2c6487e19af2e88c60668e62b778b3b0a2e70480946c1b019a"}
Nov 03 16:22:43.255408 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.255267    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"
Nov 03 16:22:43.264476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.264190    4964 scope.go:122] "RemoveContainer" containerID="1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd"
Nov 03 16:22:43.264608 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:43.264586    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd\": container with ID starting with 1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd not found: ID does not exist" containerID="1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd"
Nov 03 16:22:43.264691 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.264621    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd"} err="failed to get container status \"1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd\": rpc error: code = NotFound desc = could not find container \"1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd\": container with ID starting with 1a2f9845d88def0239cf65290ae97cfdb169b9fecdacca625a4458bf27bac2bd not found: ID does not exist"
Nov 03 16:22:43.264691 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.264645    4964 scope.go:122] "RemoveContainer" containerID="f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9"
Nov 03 16:22:43.276415 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.276358    4964 scope.go:122] "RemoveContainer" containerID="f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9"
Nov 03 16:22:43.276735 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:43.276713    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9\": container with ID starting with f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9 not found: ID does not exist" containerID="f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9"
Nov 03 16:22:43.276816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.276743    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9"} err="failed to get container status \"f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9\": rpc error: code = NotFound desc = could not find container \"f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9\": container with ID starting with f45977a9cd74d2069173c07376815cc87b4f4c52200e80c02fa5aa776a6737f9 not found: ID does not exist"
Nov 03 16:22:43.276816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.276762    4964 scope.go:122] "RemoveContainer" containerID="1881b9d20f65cd33e2ab25024197afb2cedc5b77e11dfe777daeee311bbfb5fd"
Nov 03 16:22:43.288248 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.288209    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"]
Nov 03 16:22:43.298010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.297979    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-gf8rb"]
Nov 03 16:22:43.314493 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.314441    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"]
Nov 03 16:22:43.317930 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.317902    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-xl998"]
Nov 03 16:22:43.331893 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.331867    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"]
Nov 03 16:22:43.336806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:43.336779    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-tv8zm"]
Nov 03 16:22:44.609979 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:44.609933    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" path="/var/lib/kubelet/pods/15070bd9-7432-4a73-80c0-f9cd4af75282/volumes"
Nov 03 16:22:44.610566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:44.610548    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246d37ae-a202-4576-9c03-f041c43410df" path="/var/lib/kubelet/pods/246d37ae-a202-4576-9c03-f041c43410df/volumes"
Nov 03 16:22:44.611105 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:44.611090    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b68376-e690-403a-a38b-52897ffaf73a" path="/var/lib/kubelet/pods/89b68376-e690-403a-a38b-52897ffaf73a/volumes"
Nov 03 16:22:45.309869 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:45.309834    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:47.303778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:47.303743    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"]
Nov 03 16:22:47.304168 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:47.303934    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy" containerID="cri-o://5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb" gracePeriod=30
Nov 03 16:22:47.342428 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:47.342396    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"]
Nov 03 16:22:47.342617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:47.342601    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" containerID="cri-o://8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c" gracePeriod=30
Nov 03 16:22:47.356377 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:47.356345    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"]
Nov 03 16:22:47.356599 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:47.356562    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy" containerID="cri-o://d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24" gracePeriod=30
Nov 03 16:22:49.267713 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:49.267676    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:22:49.268094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:49.267731    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:22:49.780307 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:49.780268    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:22:52.680414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.680365    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:22:52.797279 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.797249    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:22:52.844813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.844279    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:22:52.878241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878207    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878297    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878528 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878505    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.878573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878514    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.878573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878563    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878778 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878763    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.878831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878790    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.878831 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878810    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878841    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.878901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878863    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.878901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878885    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878916    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878932    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.878987 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878954    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.879073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.878992    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.879073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879023    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.879073 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879057    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.879164 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879075    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.879164 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879104    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy\") pod \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\" (UID: \"5073047c-0a5c-4f52-9be7-3c6c55c689b8\") "
Nov 03 16:22:52.879164 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879132    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.879164 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879153    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.879414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879372    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.879414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879404    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.879680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879659    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.879931 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.879918    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.880679 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.880164    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.880679 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.880431    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.881354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.881074    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.881896 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.881633    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.881896 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.881667    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "kube-api-access-gnm6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.882369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882087    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.882369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882143    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.882369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882336    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.882699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882408    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "kube-api-access-sqcq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.882699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882451    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.882786 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882718    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:22:52.882927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.882904    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:52.883844 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.883818    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo" pod "5073047c-0a5c-4f52-9be7-3c6c55c689b8" (UID: "5073047c-0a5c-4f52-9be7-3c6c55c689b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:52.884066 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.884043    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:22:52.884423 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.884368    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "kube-api-access-q4p4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.979947 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.979846    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.979947 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.979898    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.979947 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.979930    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.980249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.979966    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.980249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.979994    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.980249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980025    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy\") pod \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\" (UID: \"2ae60f3d-5fff-427c-8917-ecc52cc8929b\") "
Nov 03 16:22:52.980249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980051    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.980249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980079    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data\") pod \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\" (UID: \"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7\") "
Nov 03 16:22:52.980249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980189    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980406    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980470    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980493    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980509    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980523    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnm6n\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-kube-api-access-gnm6n\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980538    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980550    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980565    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980579    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqcq4\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-kube-api-access-sqcq4\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.980589 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980595    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980595    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980582    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980611    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980651    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980667    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980681    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980695    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980709    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980723    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073047c-0a5c-4f52-9be7-3c6c55c689b8-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980736    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.981106 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.980751    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4p4g\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-kube-api-access-q4p4g\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:52.982531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.982492    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:52.982668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.982563    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.982732 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.982700    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy" pod "2ae60f3d-5fff-427c-8917-ecc52cc8929b" (UID: "2ae60f3d-5fff-427c-8917-ecc52cc8929b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:22:52.982905 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:52.982885    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token" pod "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" (UID: "f01b1dbb-80b6-4563-9cd4-d9c3583d45e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:22:53.081156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081110    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.081156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081148    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.081156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081157    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.081156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081167    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.081459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081178    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae60f3d-5fff-427c-8917-ecc52cc8929b-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.081459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081187    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.081459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.081196    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:22:53.360174 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.360073    4964 generic.go:356] "Generic (PLEG): container finished" podID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerID="5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb" exitCode=0
Nov 03 16:22:53.360356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.360210    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" event={"ID":"2ae60f3d-5fff-427c-8917-ecc52cc8929b","Type":"ContainerDied","Data":"5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb"}
Nov 03 16:22:53.360356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.360303    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj" event={"ID":"2ae60f3d-5fff-427c-8917-ecc52cc8929b","Type":"ContainerDied","Data":"91737a807b312ed0933475b3497fc2ae48849f35bb1cfb011f046a8a0edec46e"}
Nov 03 16:22:53.360356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.360339    4964 scope.go:122] "RemoveContainer" containerID="5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb"
Nov 03 16:22:53.360588 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.360537    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"
Nov 03 16:22:53.372044 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.372012    4964 generic.go:356] "Generic (PLEG): container finished" podID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerID="8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c" exitCode=0
Nov 03 16:22:53.372144 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.372080    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"
Nov 03 16:22:53.372144 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.372094    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" event={"ID":"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7","Type":"ContainerDied","Data":"8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c"}
Nov 03 16:22:53.372246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.372148    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c" event={"ID":"f01b1dbb-80b6-4563-9cd4-d9c3583d45e7","Type":"ContainerDied","Data":"b490bc0b11e73544ab9157930504ac7461ce291e06543311387fb514a76783aa"}
Nov 03 16:22:53.374858 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.374824    4964 generic.go:356] "Generic (PLEG): container finished" podID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerID="d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24" exitCode=0
Nov 03 16:22:53.374993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.374910    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" event={"ID":"5073047c-0a5c-4f52-9be7-3c6c55c689b8","Type":"ContainerDied","Data":"d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24"}
Nov 03 16:22:53.374993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.374958    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw" event={"ID":"5073047c-0a5c-4f52-9be7-3c6c55c689b8","Type":"ContainerDied","Data":"e7e925b5af0a1e15eff29133a598cc0b82fa9403740498a95305eb56be8a1a61"}
Nov 03 16:22:53.375107 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.375033    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"
Nov 03 16:22:53.393371 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.393351    4964 scope.go:122] "RemoveContainer" containerID="5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb"
Nov 03 16:22:53.393753 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:53.393733    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb\": container with ID starting with 5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb not found: ID does not exist" containerID="5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb"
Nov 03 16:22:53.393798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.393764    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb"} err="failed to get container status \"5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb\": rpc error: code = NotFound desc = could not find container \"5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb\": container with ID starting with 5865af1aa176dbb78283c183c982b69d7b6227e9046afef102cae42b291f00bb not found: ID does not exist"
Nov 03 16:22:53.393798 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.393783    4964 scope.go:122] "RemoveContainer" containerID="8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c"
Nov 03 16:22:53.420132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.403261    4964 scope.go:122] "RemoveContainer" containerID="8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c"
Nov 03 16:22:53.420132 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:53.403666    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c\": container with ID starting with 8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c not found: ID does not exist" containerID="8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c"
Nov 03 16:22:53.420132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.403699    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c"} err="failed to get container status \"8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c\": rpc error: code = NotFound desc = could not find container \"8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c\": container with ID starting with 8f60da3dc19fd2fc2987daf582b5ee70240d5b0724e7eed7cab8ecb3aa788d3c not found: ID does not exist"
Nov 03 16:22:53.420132 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.403721    4964 scope.go:122] "RemoveContainer" containerID="d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24"
Nov 03 16:22:53.422100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.422077    4964 scope.go:122] "RemoveContainer" containerID="d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24"
Nov 03 16:22:53.422740 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:53.422667    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24\": container with ID starting with d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24 not found: ID does not exist" containerID="d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24"
Nov 03 16:22:53.422740 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.422708    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24"} err="failed to get container status \"d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24\": rpc error: code = NotFound desc = could not find container \"d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24\": container with ID starting with d60e6e6a417652fda8d722370f2c83f705aeb58bfe56de69fd9bf5c1cf931b24 not found: ID does not exist"
Nov 03 16:22:53.430694 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.430657    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"]
Nov 03 16:22:53.435041 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.435005    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-m2qsw"]
Nov 03 16:22:53.457320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.457284    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"]
Nov 03 16:22:53.463182 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.463143    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-qbf5c"]
Nov 03 16:22:53.489881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.489848    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"]
Nov 03 16:22:53.495870 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:53.495838    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-cfcmj"]
Nov 03 16:22:54.607741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:54.607709    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" path="/var/lib/kubelet/pods/2ae60f3d-5fff-427c-8917-ecc52cc8929b/volumes"
Nov 03 16:22:54.608134 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:54.608121    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" path="/var/lib/kubelet/pods/5073047c-0a5c-4f52-9be7-3c6c55c689b8/volumes"
Nov 03 16:22:54.608525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:22:54.608513    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" path="/var/lib/kubelet/pods/f01b1dbb-80b6-4563-9cd4-d9c3583d45e7/volumes"
Nov 03 16:22:55.367489 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:22:55.367458    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:23:02.566105 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:02.566066    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:23:03.682842 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:03.682796    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:23:03.683265 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:03.682909    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:23:04.770028 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:04.769984    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:23:05.379761 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:05.379722    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:23:07.026886 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.026851    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"]
Nov 03 16:23:07.027312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027298    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy"
Nov 03 16:23:07.027355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027313    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy"
Nov 03 16:23:07.027355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027326    4964 cpu_manager.go:414] "containerMap: removing container" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy"
Nov 03 16:23:07.027355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027336    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy"
Nov 03 16:23:07.027355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027354    4964 cpu_manager.go:414] "containerMap: removing container" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027359    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027367    4964 cpu_manager.go:414] "containerMap: removing container" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027372    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027378    4964 cpu_manager.go:414] "containerMap: removing container" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027398    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027410    4964 cpu_manager.go:414] "containerMap: removing container" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027416    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027429    4964 cpu_manager.go:414] "containerMap: removing container" podUID="f8882c3a-731c-4618-9243-3b1eed5606fa" containerName="revoke-keys"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027437    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8882c3a-731c-4618-9243-3b1eed5606fa" containerName="revoke-keys"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027451    4964 cpu_manager.go:414] "containerMap: removing container" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027456    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027462    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027467    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027475    4964 cpu_manager.go:414] "containerMap: removing container" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027479    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027486    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027490    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027499    4964 cpu_manager.go:414] "containerMap: removing container" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027507    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027516    4964 cpu_manager.go:414] "containerMap: removing container" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027520    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy"
Nov 03 16:23:07.027527 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027526    4964 cpu_manager.go:414] "containerMap: removing container" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027559    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027638    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5073047c-0a5c-4f52-9be7-3c6c55c689b8" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027649    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe45f54-0726-4d25-a8c5-1f582fa55534" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027660    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b68376-e690-403a-a38b-52897ffaf73a" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027668    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b920e4-9c84-4f35-981e-1a95da93e099" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027672    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01b1dbb-80b6-4563-9cd4-d9c3583d45e7" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027677    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="15070bd9-7432-4a73-80c0-f9cd4af75282" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027681    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8882c3a-731c-4618-9243-3b1eed5606fa" containerName="revoke-keys"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027687    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a800b84-3440-4c9a-9084-5ff793a7082a" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027693    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6a6bd4-9ace-4edd-9ded-a0eb4ce04469" containerName="main"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027700    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="246d37ae-a202-4576-9c03-f041c43410df" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027705    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9c1550-4b46-4a0f-b6e7-771f8691dcf2" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027710    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="774f3575-ea70-41a8-92bd-df711b8eb36c" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027716    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="242af026-10c3-4aba-baec-ef113d93d527" containerName="istio-proxy"
Nov 03 16:23:07.028175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.027721    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae60f3d-5fff-427c-8917-ecc52cc8929b" containerName="istio-proxy"
Nov 03 16:23:07.031171 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.031153    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.046253 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.046216    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"]
Nov 03 16:23:07.104531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104499    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104534    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104568    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104714    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104745    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lkhf\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104793 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104765    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104816    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104922    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.104971 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.104940    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206340 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206285    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206360    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206420    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206463    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206489    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206538    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206615    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206640    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lkhf\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206668    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.206880 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206861    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.207152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.206925    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.207209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.207184    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.207248 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.207202    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.207282 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.207245    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.209370 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.209342    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.209490 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.209474    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.219803 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.219782    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lkhf\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.219998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.219982    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-blk7t\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.345432 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.345293    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:07.635843 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.635815    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"]
Nov 03 16:23:07.636544 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:23:07.636523    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c17b7d3_4dc6_434f_9273_abdb04999618.slice/crio-25386164d2ccb4ec1bb2f67cc119c84bc1bba4a6cc67eaf189327b27b21ea43c WatchSource:0}: Error finding container 25386164d2ccb4ec1bb2f67cc119c84bc1bba4a6cc67eaf189327b27b21ea43c: Status 404 returned error can't find the container with id 25386164d2ccb4ec1bb2f67cc119c84bc1bba4a6cc67eaf189327b27b21ea43c
Nov 03 16:23:07.638791 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.638760    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:23:07.638874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.638822    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:23:07.638874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:07.638851    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:23:08.484506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:08.484460    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" event={"ID":"6c17b7d3-4dc6-434f-9273-abdb04999618","Type":"ContainerStarted","Data":"adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920"}
Nov 03 16:23:08.484899 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:08.484532    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" event={"ID":"6c17b7d3-4dc6-434f-9273-abdb04999618","Type":"ContainerStarted","Data":"25386164d2ccb4ec1bb2f67cc119c84bc1bba4a6cc67eaf189327b27b21ea43c"}
Nov 03 16:23:08.528420 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:08.528339    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podStartSLOduration=2.528320579 podStartE2EDuration="2.528320579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:08.527866754 +0000 UTC m=+1250.041248534" watchObservedRunningTime="2026-11-03 16:23:08.528320579 +0000 UTC m=+1250.041702355"
Nov 03 16:23:09.345918 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:09.345872    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:09.347280 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:09.347254    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 16:23:09.347402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:09.347304    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 16:23:10.345947 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:10.345910    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 16:23:10.346352 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:10.345959    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 16:23:11.345813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:11.345773    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 16:23:11.345996 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:11.345830    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 16:23:12.346697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:12.346655    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 16:23:12.347135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:12.346723    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 16:23:13.346681 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:13.346640    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 16:23:13.346873 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:13.346695    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 16:23:14.345751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:14.345713    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 16:23:14.345968 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:14.345793    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 16:23:15.395905 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:15.395863    4964 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ce432_ab27_4dbc_9ea2_10f933b38198.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 16:23:16.347285 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:16.346846    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:16.347285 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:16.346918    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:17.388574 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:17.388531    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:23:17.388961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:17.388586    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:23:18.358322 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:18.358291    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:18.358719 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:18.358695    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:18.359596 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:18.359572    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:23:21.384582 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:21.384552    4964 scope.go:122] "RemoveContainer" containerID="1106fe7f3e3c80981cd955440b87cb32806a0c316e40e38485b11d27e20ddc29"
Nov 03 16:23:24.034935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:24.034897    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"]
Nov 03 16:23:24.038560 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:24.038530    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d64f441-5g7zm"]
Nov 03 16:23:24.607291 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:24.607257    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e9c7d3-c760-4427-9e04-5eef25842147" path="/var/lib/kubelet/pods/d4e9c7d3-c760-4427-9e04-5eef25842147/volumes"
Nov 03 16:23:32.567049 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:23:32.567010    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:23:34.036160 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:34.036114    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"]
Nov 03 16:23:34.039486 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:34.039454    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a02195-s4dlw"]
Nov 03 16:23:34.608050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:34.608001    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6931c64c-eeb8-4226-8546-02f9d7c69de5" path="/var/lib/kubelet/pods/6931c64c-eeb8-4226-8546-02f9d7c69de5/volumes"
Nov 03 16:23:43.027258 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:43.027204    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"]
Nov 03 16:23:43.030156 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:43.030133    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144a104-rbxrh"]
Nov 03 16:23:44.608425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:44.608368    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9312ed9d-5b54-4292-847b-91866ccd15af" path="/var/lib/kubelet/pods/9312ed9d-5b54-4292-847b-91866ccd15af/volumes"
Nov 03 16:23:46.025050 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:46.025002    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"]
Nov 03 16:23:46.028343 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:46.028308    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5be5fa6-dgckb"]
Nov 03 16:23:46.607698 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:46.607654    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27107370-b018-4f71-a6b0-e3266680fe97" path="/var/lib/kubelet/pods/27107370-b018-4f71-a6b0-e3266680fe97/volumes"
Nov 03 16:23:49.024883 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:49.024841    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"]
Nov 03 16:23:49.030561 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:49.030525    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faecbd7a-87mx9"]
Nov 03 16:23:50.607546 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:50.607512    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e0c923-8445-4bf3-ad66-81f535b1f988" path="/var/lib/kubelet/pods/43e0c923-8445-4bf3-ad66-81f535b1f988/volumes"
Nov 03 16:23:53.027753 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:53.027706    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"]
Nov 03 16:23:53.030740 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:53.030718    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93522fa7-7zv8q"]
Nov 03 16:23:54.607480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:54.607444    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090de332-26a4-4c2d-99cc-4e64d72f9be9" path="/var/lib/kubelet/pods/090de332-26a4-4c2d-99cc-4e64d72f9be9/volumes"
Nov 03 16:23:58.025868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:58.025831    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"]
Nov 03 16:23:58.032266 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:58.032238    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3fe18c9-b4pst"]
Nov 03 16:23:58.607358 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:58.607313    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58176a27-d74c-4995-a91d-efb928cf6891" path="/var/lib/kubelet/pods/58176a27-d74c-4995-a91d-efb928cf6891/volumes"
Nov 03 16:23:59.024640 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:59.024606    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"]
Nov 03 16:23:59.029191 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:23:59.029154    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3754ce8f-rptkz"]
Nov 03 16:24:00.608129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:00.608096    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09daff1a-856e-4756-aa29-cef67c983318" path="/var/lib/kubelet/pods/09daff1a-856e-4756-aa29-cef67c983318/volumes"
Nov 03 16:24:02.009967 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:02.009930    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:24:02.581833 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:24:02.581778    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:24:03.027409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:03.027340    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"]
Nov 03 16:24:03.031887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:03.031861    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6512d20d-l9jhg"]
Nov 03 16:24:04.406413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:04.402606    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-blk7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:24:04.406413 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:04.402689    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:24:04.625533 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:04.625495    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a817b7a-388a-435f-b539-31e6b936d957" path="/var/lib/kubelet/pods/6a817b7a-388a-435f-b539-31e6b936d957/volumes"
Nov 03 16:24:09.024774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:09.024739    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"]
Nov 03 16:24:09.027480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:09.027453    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9514f47-899kg"]
Nov 03 16:24:10.608109 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:10.608070    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0987bc-8d70-4214-b013-e49251b1e5a3" path="/var/lib/kubelet/pods/eb0987bc-8d70-4214-b013-e49251b1e5a3/volumes"
Nov 03 16:24:11.028565 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:11.028526    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"]
Nov 03 16:24:11.033825 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:11.033794    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a772b6e-kl4sh"]
Nov 03 16:24:12.607905 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:12.607870    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2" path="/var/lib/kubelet/pods/ce3aa72a-bfb0-4a66-af13-b59eff1dd4e2/volumes"
Nov 03 16:24:14.029902 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:14.029862    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"]
Nov 03 16:24:14.032636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:14.032605    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ec88da-nfgkg"]
Nov 03 16:24:14.608366 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:14.608329    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20080c8e-ebd5-45b8-a274-4bfdfe08fe80" path="/var/lib/kubelet/pods/20080c8e-ebd5-45b8-a274-4bfdfe08fe80/volumes"
Nov 03 16:24:20.392089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:20.392056    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:24:21.484832 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:21.484799    4964 scope.go:122] "RemoveContainer" containerID="3e6c38794abda890dac7bc5708548b351c7e9f8cd0f38c424c30a9aa67c9a4c7"
Nov 03 16:24:25.039411 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:25.034536    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"]
Nov 03 16:24:25.041683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:25.040932    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970c6349-st6gl"]
Nov 03 16:24:25.300320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:25.300239    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:24:26.024350 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:26.024311    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"]
Nov 03 16:24:26.026929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:26.026900    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036eb5f2-gtr8v"]
Nov 03 16:24:26.609424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:26.609362    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10783943-763b-4d29-ba95-ad148302c624" path="/var/lib/kubelet/pods/10783943-763b-4d29-ba95-ad148302c624/volumes"
Nov 03 16:24:26.609993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:26.609975    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a257de9-1680-44a9-9acd-42eee14feb05" path="/var/lib/kubelet/pods/5a257de9-1680-44a9-9acd-42eee14feb05/volumes"
Nov 03 16:24:32.566424 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:24:32.566368    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:24:36.034788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:36.034757    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"]
Nov 03 16:24:36.037590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:36.037564    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558f91e9-rfpbl"]
Nov 03 16:24:36.607949 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:36.607916    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60558c0e-271d-48fe-b286-1a39aaa92498" path="/var/lib/kubelet/pods/60558c0e-271d-48fe-b286-1a39aaa92498/volumes"
Nov 03 16:24:41.300704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:41.300662    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:24:46.066558 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:46.066509    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:24:46.067491 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:46.067451    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:24:46.193617 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:46.193572    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:24:46.193825 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:46.193652    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:24:54.089303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:24:54.089261    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:25:01.026765 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:01.026729    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"]
Nov 03 16:25:01.030281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:01.030257    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b34fdc2-l94vq"]
Nov 03 16:25:02.565797 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:25:02.565758    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:25:02.608315 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:02.608281    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b588ca2-7a14-4690-a6eb-2b0ffcc8640c" path="/var/lib/kubelet/pods/2b588ca2-7a14-4690-a6eb-2b0ffcc8640c/volumes"
Nov 03 16:25:18.026213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:18.026168    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"]
Nov 03 16:25:18.029072 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:18.029038    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55985def-x7hsk"]
Nov 03 16:25:18.488892 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:18.488854    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:25:18.608204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:18.608166    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8192ee-f489-4aff-9b92-a4b476f61ca1" path="/var/lib/kubelet/pods/db8192ee-f489-4aff-9b92-a4b476f61ca1/volumes"
Nov 03 16:25:21.518219 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.518189    4964 scope.go:122] "RemoveContainer" containerID="6cb6e9f9f456e9d92fb663fa0abd642c073855f16db208afaf0fda39faa191c1"
Nov 03 16:25:21.527603 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.527573    4964 scope.go:122] "RemoveContainer" containerID="77d361434ee1a08771901288975bd03a10721489486b36170118b4c2144970ce"
Nov 03 16:25:21.536256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.536240    4964 scope.go:122] "RemoveContainer" containerID="397e2c63bd1ec65a1f011bb999fbbf4b357a6623af977719f02cc29a41665078"
Nov 03 16:25:21.545614 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.545591    4964 scope.go:122] "RemoveContainer" containerID="80021295ec652e6fd841275d06aa8af627bc9a4d5c74bce68c51828945c33da2"
Nov 03 16:25:21.553771 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.553752    4964 scope.go:122] "RemoveContainer" containerID="b8e60459c785c2c3ca14b89f42340f5c363e128d2fc72059c167a3470810266e"
Nov 03 16:25:21.562149 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.562127    4964 scope.go:122] "RemoveContainer" containerID="e1833f14c12893ad5690496354a800c8c74db2b78ead49a068084811f73c6b7b"
Nov 03 16:25:21.571161 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.571138    4964 scope.go:122] "RemoveContainer" containerID="fd650dcfa70f38b06157cf1edceef09e7e4b2b94b001f0164d265f2eb28a3f11"
Nov 03 16:25:21.579966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.579938    4964 scope.go:122] "RemoveContainer" containerID="0f77c7152ed0e7ef75844b11d761803756a4fe6cd629214df66c23d784c14091"
Nov 03 16:25:21.588945 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.588924    4964 scope.go:122] "RemoveContainer" containerID="87d366ddcabb050ae01c1db56bd0ddf3c2eb7fa9fb9ff414b4bea25046140ff5"
Nov 03 16:25:21.598930 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.598909    4964 scope.go:122] "RemoveContainer" containerID="846eeb2e6b41d7375f412fcaaa2e3662398033af926327eeac263a4d984cf2e4"
Nov 03 16:25:21.609468 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:21.609443    4964 scope.go:122] "RemoveContainer" containerID="e492b7dcea368e0b19381ed485e0fce3585589adf48d45877939559233c0ce49"
Nov 03 16:25:28.686860 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:28.686813    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:25:32.626411 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:25:32.624689    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:25:33.687756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:33.687713    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:25:34.042471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:34.042372    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"]
Nov 03 16:25:34.046633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:34.046603    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c76c0-rrjwg"]
Nov 03 16:25:34.607671 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:34.607633    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0230724-c182-423c-bca6-542e675034bc" path="/var/lib/kubelet/pods/a0230724-c182-423c-bca6-542e675034bc/volumes"
Nov 03 16:25:45.024711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:45.024676    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"]
Nov 03 16:25:45.029175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:45.029141    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97434216-75dxk"]
Nov 03 16:25:46.608103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:46.608058    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5065088f-eec5-4e2a-81af-689a0a58972e" path="/var/lib/kubelet/pods/5065088f-eec5-4e2a-81af-689a0a58972e/volumes"
Nov 03 16:25:49.491884 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:49.491835    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:25:52.026090 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:52.026054    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"]
Nov 03 16:25:52.028871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:52.028837    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397f76bc-rz7zk"]
Nov 03 16:25:52.618102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:52.618061    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78310d4-be52-437f-8059-e2ea0b2c6564" path="/var/lib/kubelet/pods/b78310d4-be52-437f-8059-e2ea0b2c6564/volumes"
Nov 03 16:25:53.327757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:53.327703    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:25:59.039244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:59.039200    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"]
Nov 03 16:25:59.044288 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:25:59.044259    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2909aa-c7wq2"]
Nov 03 16:26:00.607569 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:00.607537    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cb7287-9039-47a4-9afa-e3f310eb7c26" path="/var/lib/kubelet/pods/76cb7287-9039-47a4-9afa-e3f310eb7c26/volumes"
Nov 03 16:26:02.566867 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:26:02.566820    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:26:07.814876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:07.814830    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:26:08.129859 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:08.129777    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"]
Nov 03 16:26:08.138939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:08.138906    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f9fd18-fbcxb"]
Nov 03 16:26:08.608560 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:08.608522    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6caf06c-371a-4c97-b999-01a057bf3f8b" path="/var/lib/kubelet/pods/d6caf06c-371a-4c97-b999-01a057bf3f8b/volumes"
Nov 03 16:26:13.797913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:13.797874    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:26:21.041999 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.041957    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"]
Nov 03 16:26:21.047620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.047569    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c89d8ba-cccn2"]
Nov 03 16:26:21.688046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.688009    4964 scope.go:122] "RemoveContainer" containerID="e3a0e1d348ea391ff55f08944fbd2e13ba1621aff0bcbc945bb218bbf3a9e735"
Nov 03 16:26:21.697584 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.697562    4964 scope.go:122] "RemoveContainer" containerID="71e91d1ef8e459bf9ae847f21f030d391395b66b1417c47bfbabdb8e1559aac9"
Nov 03 16:26:21.706675 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.706649    4964 scope.go:122] "RemoveContainer" containerID="a3fcf9f163dc1f464b8273e8b78cc076a8b9ad894fba67acdc3ab80125c4b1da"
Nov 03 16:26:21.715757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.715734    4964 scope.go:122] "RemoveContainer" containerID="eca5950ab259f88de1aeca1dda6f86df6d18b02613c736408b631c28f8cb77ea"
Nov 03 16:26:21.724666 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:21.724647    4964 scope.go:122] "RemoveContainer" containerID="692cdc71a249c2712e5399da2829f77d835341d7f0a6456c3d19da2e3d6e4356"
Nov 03 16:26:22.608056 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:22.608021    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e0921b-e10b-44d0-833a-c52031d5e6d7" path="/var/lib/kubelet/pods/32e0921b-e10b-44d0-833a-c52031d5e6d7/volumes"
Nov 03 16:26:23.030894 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:23.030856    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"]
Nov 03 16:26:23.034992 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:23.034960    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e9764-kdq5f"]
Nov 03 16:26:24.608690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:24.608653    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5be3fb-4c8d-4567-8a6a-07eb64d13c04" path="/var/lib/kubelet/pods/5d5be3fb-4c8d-4567-8a6a-07eb64d13c04/volumes"
Nov 03 16:26:27.006445 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:27.006406    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:26:30.841425 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:30.841359    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:26:31.173412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:31.170500    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:26:31.173412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:31.170571    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:26:32.567178 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:26:32.567140    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:26:34.038928 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:34.038891    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"]
Nov 03 16:26:34.043508 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:34.043476    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"]
Nov 03 16:26:34.045825 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:34.045784    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93e274f6-p7vg9"]
Nov 03 16:26:34.047621 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:34.047594    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d65a089-4wgrh"]
Nov 03 16:26:34.608291 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:34.608253    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126c9c1e-9b01-427a-9b3e-878880651330" path="/var/lib/kubelet/pods/126c9c1e-9b01-427a-9b3e-878880651330/volumes"
Nov 03 16:26:34.608708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:34.608693    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbba35b9-e74c-4547-857a-16445bc1bd55" path="/var/lib/kubelet/pods/dbba35b9-e74c-4547-857a-16445bc1bd55/volumes"
Nov 03 16:26:40.031876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:40.031838    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"]
Nov 03 16:26:40.044337 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:40.044301    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63aadd61-k6t29"]
Nov 03 16:26:40.607356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:40.607312    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2f81d1-ac41-440e-96f1-bcc075ea7ae4" path="/var/lib/kubelet/pods/fb2f81d1-ac41-440e-96f1-bcc075ea7ae4/volumes"
Nov 03 16:26:44.886986 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:44.886946    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:26:53.023718 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:53.023680    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"]
Nov 03 16:26:53.027011 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:53.026974    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6f5f43-l8f5s"]
Nov 03 16:26:54.607973 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:54.607936    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b124965c-f0c9-4e4d-a73e-d8f704addbb1" path="/var/lib/kubelet/pods/b124965c-f0c9-4e4d-a73e-d8f704addbb1/volumes"
Nov 03 16:26:55.705241 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:55.705200    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:26:59.029660 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:59.029410    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"]
Nov 03 16:26:59.035213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:59.035137    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"]
Nov 03 16:26:59.038209 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:59.038176    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fccc1b-wt5n2"]
Nov 03 16:26:59.041190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:26:59.041155    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afeece82-qcfp5"]
Nov 03 16:27:00.034843 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:00.034800    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"]
Nov 03 16:27:00.038977 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:00.038925    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0227261-54zfk"]
Nov 03 16:27:00.421480 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:00.421430    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:27:00.619416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:00.618632    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d4caff-1075-4f3b-903b-8a9ac091fff2" path="/var/lib/kubelet/pods/06d4caff-1075-4f3b-903b-8a9ac091fff2/volumes"
Nov 03 16:27:00.619416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:00.619243    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39957cfb-dda9-48dc-8374-aeacffe73815" path="/var/lib/kubelet/pods/39957cfb-dda9-48dc-8374-aeacffe73815/volumes"
Nov 03 16:27:00.620429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:00.619838    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fb9651-3d02-461c-937a-20eb9241afdb" path="/var/lib/kubelet/pods/c4fb9651-3d02-461c-937a-20eb9241afdb/volumes"
Nov 03 16:27:01.067412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:01.066547    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:27:01.067412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:01.066707    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:27:01.183573 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:01.183528    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zphl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:27:01.183962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:01.183921    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zphl8" podUID="e2dc90c6-0535-4ae9-bb93-5c6ee5f68345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:27:02.567763 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:27:02.567718    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:27:17.687103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:17.687064    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:27:19.025472 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:19.025434    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"]
Nov 03 16:27:19.031306 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:19.031269    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46173954-57jtq"]
Nov 03 16:27:20.608605 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:20.608569    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844ce432-ab27-4dbc-9ea2-10f933b38198" path="/var/lib/kubelet/pods/844ce432-ab27-4dbc-9ea2-10f933b38198/volumes"
Nov 03 16:27:21.283682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.283656    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:27:21.293000 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.292975    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:27:21.681566 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.681523    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:27:21.766499 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.766465    4964 scope.go:122] "RemoveContainer" containerID="9fd83ef8c0cc93a55e568255d4ac7a3c0552b994e46aabd88a309f284aa7646e"
Nov 03 16:27:21.776310 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.776285    4964 scope.go:122] "RemoveContainer" containerID="8b0e1a9f2b5dbfa9173e24485eaa61a35a01b08a1258f53f55b21ef2e0510ca0"
Nov 03 16:27:21.786029 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.785998    4964 scope.go:122] "RemoveContainer" containerID="c82f12c6e28d02d62f42fab4e5917bab0b4eb08a2b1c086d23a2babf03e069dd"
Nov 03 16:27:21.795052 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.795025    4964 scope.go:122] "RemoveContainer" containerID="0edbccd58f3b8f8cef271e030d07b81c1e0e303b24dbabd659533bc200004153"
Nov 03 16:27:21.804474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.804444    4964 scope.go:122] "RemoveContainer" containerID="92dfdc5ae8b97ba217cc611e98f648e1bcaa310829140eca331bfaffba30a881"
Nov 03 16:27:21.816773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.816743    4964 scope.go:122] "RemoveContainer" containerID="18fd0293c44e884aa41416961a4f18750aabf623ef33fdf3e6258a49bd4b8700"
Nov 03 16:27:21.826369 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.826348    4964 scope.go:122] "RemoveContainer" containerID="dfcbd3d44b228ee44c3ecd6ff53f6a065ef67e7be903a4487dc0609e94e14258"
Nov 03 16:27:21.835342 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.835320    4964 scope.go:122] "RemoveContainer" containerID="5ab99ed62a9e81a5c3c031398bb3d7fdb75e7604c08f298ce43fc56825adb244"
Nov 03 16:27:21.844273 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:21.844250    4964 scope.go:122] "RemoveContainer" containerID="2452daea98e87432c8ffc390035c8737d0a16f77b74695598f2edfebc60ca856"
Nov 03 16:27:27.409038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:27.408994    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:27:32.025090 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:32.025049    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"]
Nov 03 16:27:32.030187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:32.030157    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed08bf69-l4tz9"]
Nov 03 16:27:32.565456 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:27:32.565415    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:27:32.607723 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:32.607685    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8882c3a-731c-4618-9243-3b1eed5606fa" path="/var/lib/kubelet/pods/f8882c3a-731c-4618-9243-3b1eed5606fa/volumes"
Nov 03 16:27:46.006772 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:46.006732    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:27:56.186170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:27:56.186129    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:28:02.566218 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:28:02.566177    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:28:21.903871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.903838    4964 scope.go:122] "RemoveContainer" containerID="7e97a1849b5e418d8be01e39b8fa283b3da5fe513a6f2dadddb720df6ca8ecbb"
Nov 03 16:28:21.914969 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.914940    4964 scope.go:122] "RemoveContainer" containerID="2da81af0724ab5b95187a68830d68688e2027a4d305d68994f5b855fe629f88c"
Nov 03 16:28:21.924141 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.924116    4964 scope.go:122] "RemoveContainer" containerID="abace5763728e49fa9c4a57a6fec57cc57aa8498d96bf5e970e2d2a230c8b766"
Nov 03 16:28:21.932970 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.932947    4964 scope.go:122] "RemoveContainer" containerID="ee22df8b2da17fb88678109fbe49a09b0d14f83efdd190d311fda50f2bb9f6b0"
Nov 03 16:28:21.941405 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.941362    4964 scope.go:122] "RemoveContainer" containerID="483c8f27c754d4ccd4d77c1f6875e9f294fd3c5691413a3bc3d081b02bc7ad4d"
Nov 03 16:28:21.949720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.949699    4964 scope.go:122] "RemoveContainer" containerID="96e2ef58dbffe6892482f17ba5121c01ecb14fbfd2cd164182a1b7cc7d9e4f4e"
Nov 03 16:28:21.958004 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.957980    4964 scope.go:122] "RemoveContainer" containerID="fd7a5901a9564e53b06d709492f2c4116133606579acee8f81851d0f1a29d6ac"
Nov 03 16:28:21.966414 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.966369    4964 scope.go:122] "RemoveContainer" containerID="15a02e25a714a67de4550afbfecc4d5727952db880596186d6115dc8f1fe2c19"
Nov 03 16:28:21.976180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.976158    4964 scope.go:122] "RemoveContainer" containerID="86f5ca7564789af6f82fe2db88c736e4535f13fdf6379c0159cbc8302a62e2a8"
Nov 03 16:28:21.984913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.984892    4964 scope.go:122] "RemoveContainer" containerID="e693efab1781a5012cce97b54710d282842f54d062901dc37a9efe33e22930bf"
Nov 03 16:28:21.993430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:28:21.993405    4964 scope.go:122] "RemoveContainer" containerID="a8b71cf6b8655487b3c030107c8adc0ceca481366062acf778d5ad44f59f2d97"
Nov 03 16:28:32.565910 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:28:32.565825    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:29:02.567439 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:29:02.567397    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:29:22.052856 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:29:22.052822    4964 scope.go:122] "RemoveContainer" containerID="b7ec7b8fc63e012227e7dafcfe409f0567c811d9eec8c5b88fa4f9af16cbf094"
Nov 03 16:29:32.564844 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:29:32.564804    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:30:02.566672 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:30:02.566578    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:30:32.565955 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:30:32.565908    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:31:02.566649 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:31:02.566603    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:31:32.565559 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:31:32.565478    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:32:02.566264 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:32:02.566223    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:32:21.320949 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:32:21.320918    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:32:21.326877 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:32:21.326849    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:32:32.567371 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:32:32.567333    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:33:02.566587 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:33:02.566496    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:33:07.801054 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:07.801014    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"]
Nov 03 16:33:07.801531 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:07.801216    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy" containerID="cri-o://060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb" gracePeriod=30
Nov 03 16:33:13.053068 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.053034    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:33:13.097314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097233    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097277    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097314 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097298    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097324    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097346    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097419    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097480    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097568    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.097625 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097578    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097655    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo\") pod \"5a5413eb-9a57-4209-a52f-b0c3d0408365\" (UID: \"5a5413eb-9a57-4209-a52f-b0c3d0408365\") "
Nov 03 16:33:13.097932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097754    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.097932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097774    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.098111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.097947    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.098111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.098036    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.098111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.098050    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.098111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.098038    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:13.098111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.098060    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.098111 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.098069    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.099927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.099898    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:13.099927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.099905    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "kube-api-access-d4nmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:13.100162 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.100144    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:33:13.100482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.100456    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token" pod "5a5413eb-9a57-4209-a52f-b0c3d0408365" (UID: "5a5413eb-9a57-4209-a52f-b0c3d0408365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:13.198995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.198951    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.198995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.198990    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5413eb-9a57-4209-a52f-b0c3d0408365-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199006    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199021    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5413eb-9a57-4209-a52f-b0c3d0408365-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199036    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nmt\" (UniqueName: \"kubernetes.io/projected/5a5413eb-9a57-4209-a52f-b0c3d0408365-kube-api-access-d4nmt\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199107    4964 generic.go:356] "Generic (PLEG): container finished" podID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerID="060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb" exitCode=0
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199154    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" event={"ID":"5a5413eb-9a57-4209-a52f-b0c3d0408365","Type":"ContainerDied","Data":"060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb"}
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199184    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"
Nov 03 16:33:13.199249 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199243    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg" event={"ID":"5a5413eb-9a57-4209-a52f-b0c3d0408365","Type":"ContainerDied","Data":"bbb082137e5dca5d101b1c26d9269cb2fb8894fed37869b9556ec4787976185f"}
Nov 03 16:33:13.199610 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.199274    4964 scope.go:122] "RemoveContainer" containerID="060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb"
Nov 03 16:33:13.209017 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.208987    4964 scope.go:122] "RemoveContainer" containerID="060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb"
Nov 03 16:33:13.209329 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:33:13.209305    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb\": container with ID starting with 060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb not found: ID does not exist" containerID="060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb"
Nov 03 16:33:13.209437 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.209339    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb"} err="failed to get container status \"060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb\": rpc error: code = NotFound desc = could not find container \"060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb\": container with ID starting with 060edb27ab9fcbfdfba89f9d805c328e36cea4e00079e0acf786835ffd8bdfdb not found: ID does not exist"
Nov 03 16:33:13.276460 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.276425    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"]
Nov 03 16:33:13.293069 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:13.293029    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7z9tg"]
Nov 03 16:33:14.609665 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:14.609618    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" path="/var/lib/kubelet/pods/5a5413eb-9a57-4209-a52f-b0c3d0408365/volumes"
Nov 03 16:33:15.803588 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:15.803543    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:33:20.194226 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:20.194180    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:33:22.831680 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:22.831630    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"]
Nov 03 16:33:22.832208 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:22.831862    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy" containerID="cri-o://adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920" gracePeriod=30
Nov 03 16:33:28.104046 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.104016    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:33:28.275962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.275917    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.275989    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276031    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276067    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276105    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276137    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276196    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276256    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276298    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf\") pod \"6c17b7d3-4dc6-434f-9273-abdb04999618\" (UID: \"6c17b7d3-4dc6-434f-9273-abdb04999618\") "
Nov 03 16:33:28.276494 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276413    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.276691 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.276645    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-certs\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.277083 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.277016    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.277083 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.277035    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.277263 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.277220    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:28.277340 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.277280    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.278593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.278560    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:28.278788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.278764    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "kube-api-access-6lkhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:28.278889 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.278864    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:28.279182 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.279161    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo" pod "6c17b7d3-4dc6-434f-9273-abdb04999618" (UID: "6c17b7d3-4dc6-434f-9273-abdb04999618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:33:28.295968 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.295933    4964 generic.go:356] "Generic (PLEG): container finished" podID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerID="adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920" exitCode=0
Nov 03 16:33:28.296142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.295985    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"
Nov 03 16:33:28.296142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.296013    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" event={"ID":"6c17b7d3-4dc6-434f-9273-abdb04999618","Type":"ContainerDied","Data":"adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920"}
Nov 03 16:33:28.296142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.296071    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t" event={"ID":"6c17b7d3-4dc6-434f-9273-abdb04999618","Type":"ContainerDied","Data":"25386164d2ccb4ec1bb2f67cc119c84bc1bba4a6cc67eaf189327b27b21ea43c"}
Nov 03 16:33:28.296142 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.296093    4964 scope.go:122] "RemoveContainer" containerID="adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920"
Nov 03 16:33:28.308930 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.308881    4964 scope.go:122] "RemoveContainer" containerID="adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920"
Nov 03 16:33:28.309312 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:33:28.309282    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920\": container with ID starting with adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920 not found: ID does not exist" containerID="adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920"
Nov 03 16:33:28.309467 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.309326    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920"} err="failed to get container status \"adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920\": rpc error: code = NotFound desc = could not find container \"adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920\": container with ID starting with adf04b01e5929104059f675473057f081fe75187cb700651321217fe3ae0f920 not found: ID does not exist"
Nov 03 16:33:28.337661 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.337625    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"]
Nov 03 16:33:28.350660 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.350629    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-blk7t"]
Nov 03 16:33:28.377289 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377250    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-token\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377289 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377278    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-envoy\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377289 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377290    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-workload-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377300    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-credential-socket\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377310    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-data\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377319    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17b7d3-4dc6-434f-9273-abdb04999618-istiod-ca-cert\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377329    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17b7d3-4dc6-434f-9273-abdb04999618-istio-podinfo\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.377593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.377339    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lkhf\" (UniqueName: \"kubernetes.io/projected/6c17b7d3-4dc6-434f-9273-abdb04999618-kube-api-access-6lkhf\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:28.610576 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:28.610497    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" path="/var/lib/kubelet/pods/6c17b7d3-4dc6-434f-9273-abdb04999618/volumes"
Nov 03 16:33:30.438606 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:30.438566    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:33:32.567795 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:33:32.567750    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:33:41.311768 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:41.311711    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:33:57.210250 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:33:57.210206    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:34:02.566737 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:34:02.566699    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:34:09.056170 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056128    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"]
Nov 03 16:34:09.056636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056583    4964 cpu_manager.go:414] "containerMap: removing container" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy"
Nov 03 16:34:09.056636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056596    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy"
Nov 03 16:34:09.056636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056614    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy"
Nov 03 16:34:09.056636 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056620    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy"
Nov 03 16:34:09.056770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056701    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c17b7d3-4dc6-434f-9273-abdb04999618" containerName="istio-proxy"
Nov 03 16:34:09.056770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.056715    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5413eb-9a57-4209-a52f-b0c3d0408365" containerName="istio-proxy"
Nov 03 16:34:09.060085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.060067    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.064055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.064018    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:34:09.064055 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.064023    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:34:09.065066 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.065037    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:34:09.065175 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.065104    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:34:09.068881 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.068854    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"]
Nov 03 16:34:09.171685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.171647    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.171887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.171696    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7km\" (UniqueName: \"kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.171887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.171808    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.272594 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.272564    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.274139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.272610    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7km\" (UniqueName: \"kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.274139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.272679    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.274139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.273485    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.275993 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.275915    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.286324 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.286296    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7km\" (UniqueName: \"kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km\") pod \"maas-api-revoke-sub-b81b3e80-l4js9\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.375082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.374995    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:09.661656 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:09.661625    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"]
Nov 03 16:34:09.664590 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:34:09.664546    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfb7717_edd5_4eff_b95d_ab52a1e7c1ed.slice/crio-ad7c9dc7c3a04df1da3daed24919ab8584c849ab3ee20d3558d03791a19d3836 WatchSource:0}: Error finding container ad7c9dc7c3a04df1da3daed24919ab8584c849ab3ee20d3558d03791a19d3836: Status 404 returned error can't find the container with id ad7c9dc7c3a04df1da3daed24919ab8584c849ab3ee20d3558d03791a19d3836
Nov 03 16:34:10.514090 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:10.514052    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9" event={"ID":"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed","Type":"ContainerStarted","Data":"7f6b8315cb65d36467c9c6f46225606d4221984ff22cb469b79710ea9660e779"}
Nov 03 16:34:10.514545 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:10.514116    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9" event={"ID":"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed","Type":"ContainerStarted","Data":"ad7c9dc7c3a04df1da3daed24919ab8584c849ab3ee20d3558d03791a19d3836"}
Nov 03 16:34:10.535243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:10.535188    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9" podStartSLOduration=1.5351744630000002 podStartE2EDuration="1.535174463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:10.533656488 +0000 UTC m=+1912.047038265" watchObservedRunningTime="2026-11-03 16:34:10.535174463 +0000 UTC m=+1912.048556239"
Nov 03 16:34:11.526673 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:11.526640    4964 generic.go:356] "Generic (PLEG): container finished" podID="3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" containerID="7f6b8315cb65d36467c9c6f46225606d4221984ff22cb469b79710ea9660e779" exitCode=0
Nov 03 16:34:11.527071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:11.526731    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9" event={"ID":"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed","Type":"ContainerDied","Data":"7f6b8315cb65d36467c9c6f46225606d4221984ff22cb469b79710ea9660e779"}
Nov 03 16:34:12.676632 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.676605    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:12.803936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.803849    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km\" (UniqueName: \"kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km\") pod \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") "
Nov 03 16:34:12.804096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.803962    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca\") pod \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") "
Nov 03 16:34:12.804096 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.803988    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca\") pod \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\" (UID: \"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed\") "
Nov 03 16:34:12.804447 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.804420    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca" pod "3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" (UID: "3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:34:12.806118 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.806090    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca" pod "3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" (UID: "3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:34:12.806204 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.806101    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km" pod "3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" (UID: "3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed"). InnerVolumeSpecName "kube-api-access-tq7km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:34:12.904806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.904769    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:12.904806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.904804    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:12.905023 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:12.904819    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7km\" (UniqueName: \"kubernetes.io/projected/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed-kube-api-access-tq7km\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:13.537783 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:13.537746    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9" event={"ID":"3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed","Type":"ContainerDied","Data":"ad7c9dc7c3a04df1da3daed24919ab8584c849ab3ee20d3558d03791a19d3836"}
Nov 03 16:34:13.537783 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:13.537774    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"
Nov 03 16:34:13.538014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:13.537798    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad7c9dc7c3a04df1da3daed24919ab8584c849ab3ee20d3558d03791a19d3836"
Nov 03 16:34:20.170246 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.170196    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"]
Nov 03 16:34:20.170874 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.170848    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" containerName="revoke-keys"
Nov 03 16:34:20.170946 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.170875    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" containerName="revoke-keys"
Nov 03 16:34:20.171024 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.171011    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" containerName="revoke-keys"
Nov 03 16:34:20.174927 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.174898    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.179974 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.179945    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:34:20.182517 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.182462    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:34:20.182668 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.182636    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:34:20.182829 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.182812    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:34:20.185991 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.185960    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"]
Nov 03 16:34:20.272633 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.272590    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.272816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.272650    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqnm\" (UniqueName: \"kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.272816 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.272801    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.374216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.374174    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.374416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.374284    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.374416 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.374323    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqnm\" (UniqueName: \"kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.375180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.375150    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.376678 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.376645    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.387430 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.387367    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqnm\" (UniqueName: \"kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.489882 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.489778    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:20.774809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.774781    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"]
Nov 03 16:34:20.777345 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:34:20.777311    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79eaeeed_263f_4d79_817a_b3d1eefe0682.slice/crio-e397118637c659c325261c6e2fbe0a05ca1566d33400a45bb8ca1471c991bdd9 WatchSource:0}: Error finding container e397118637c659c325261c6e2fbe0a05ca1566d33400a45bb8ca1471c991bdd9: Status 404 returned error can't find the container with id e397118637c659c325261c6e2fbe0a05ca1566d33400a45bb8ca1471c991bdd9
Nov 03 16:34:20.835654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:20.835602    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:34:21.578789 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:21.578747    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g" event={"ID":"79eaeeed-263f-4d79-817a-b3d1eefe0682","Type":"ContainerStarted","Data":"4816ee0c7a2c35ff0728a9737a58426c5831cc74822a660bd5210e4d8bd33d9c"}
Nov 03 16:34:21.579187 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:21.578804    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g" event={"ID":"79eaeeed-263f-4d79-817a-b3d1eefe0682","Type":"ContainerStarted","Data":"e397118637c659c325261c6e2fbe0a05ca1566d33400a45bb8ca1471c991bdd9"}
Nov 03 16:34:21.617478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:21.617412    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g" podStartSLOduration=1.61737839 podStartE2EDuration="1.61737839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:21.616570022 +0000 UTC m=+1923.129951798" watchObservedRunningTime="2026-11-03 16:34:21.61737839 +0000 UTC m=+1923.130760168"
Nov 03 16:34:22.585271 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:22.585233    4964 generic.go:356] "Generic (PLEG): container finished" podID="79eaeeed-263f-4d79-817a-b3d1eefe0682" containerID="4816ee0c7a2c35ff0728a9737a58426c5831cc74822a660bd5210e4d8bd33d9c" exitCode=0
Nov 03 16:34:22.585708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:22.585318    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g" event={"ID":"79eaeeed-263f-4d79-817a-b3d1eefe0682","Type":"ContainerDied","Data":"4816ee0c7a2c35ff0728a9737a58426c5831cc74822a660bd5210e4d8bd33d9c"}
Nov 03 16:34:23.738929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.738900    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:23.808992 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.808951    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm\" (UniqueName: \"kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm\") pod \"79eaeeed-263f-4d79-817a-b3d1eefe0682\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") "
Nov 03 16:34:23.809196 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.809092    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca\") pod \"79eaeeed-263f-4d79-817a-b3d1eefe0682\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") "
Nov 03 16:34:23.809196 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.809163    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca\") pod \"79eaeeed-263f-4d79-817a-b3d1eefe0682\" (UID: \"79eaeeed-263f-4d79-817a-b3d1eefe0682\") "
Nov 03 16:34:23.809560 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.809531    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca" pod "79eaeeed-263f-4d79-817a-b3d1eefe0682" (UID: "79eaeeed-263f-4d79-817a-b3d1eefe0682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:34:23.811354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.811328    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca" pod "79eaeeed-263f-4d79-817a-b3d1eefe0682" (UID: "79eaeeed-263f-4d79-817a-b3d1eefe0682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:34:23.811682 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.811658    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm" pod "79eaeeed-263f-4d79-817a-b3d1eefe0682" (UID: "79eaeeed-263f-4d79-817a-b3d1eefe0682"). InnerVolumeSpecName "kube-api-access-xvqnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:34:23.911027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.910984    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:23.911027 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.911027    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eaeeed-263f-4d79-817a-b3d1eefe0682-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:23.911257 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:23.911043    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqnm\" (UniqueName: \"kubernetes.io/projected/79eaeeed-263f-4d79-817a-b3d1eefe0682-kube-api-access-xvqnm\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:24.596276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:24.596244    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"
Nov 03 16:34:24.596474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:24.596242    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g" event={"ID":"79eaeeed-263f-4d79-817a-b3d1eefe0682","Type":"ContainerDied","Data":"e397118637c659c325261c6e2fbe0a05ca1566d33400a45bb8ca1471c991bdd9"}
Nov 03 16:34:24.596474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:24.596370    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e397118637c659c325261c6e2fbe0a05ca1566d33400a45bb8ca1471c991bdd9"
Nov 03 16:34:31.592884 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:31.592793    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"]
Nov 03 16:34:31.602757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:31.602727    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ba8f54-j5h8g"]
Nov 03 16:34:32.565668 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:34:32.565628    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:34:32.607923 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:32.607888    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79eaeeed-263f-4d79-817a-b3d1eefe0682" path="/var/lib/kubelet/pods/79eaeeed-263f-4d79-817a-b3d1eefe0682/volumes"
Nov 03 16:34:33.179370 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:33.179323    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:34:50.987606 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:50.987568    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:34:54.701885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:54.701839    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:34:58.589735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.589691    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"]
Nov 03 16:34:58.590256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.589935    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9" podUID="42924e66-4865-4cd3-9771-4e819f4bacb2" containerName="manager" containerID="cri-o://d15eff275e1337f773f9bd858d99fcc1d40cf2ed37e019d2dc07d9ccdebdcba5" gracePeriod=10
Nov 03 16:34:58.774048 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.774007    4964 generic.go:356] "Generic (PLEG): container finished" podID="42924e66-4865-4cd3-9771-4e819f4bacb2" containerID="d15eff275e1337f773f9bd858d99fcc1d40cf2ed37e019d2dc07d9ccdebdcba5" exitCode=0
Nov 03 16:34:58.774211 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.774084    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9" event={"ID":"42924e66-4865-4cd3-9771-4e819f4bacb2","Type":"ContainerDied","Data":"d15eff275e1337f773f9bd858d99fcc1d40cf2ed37e019d2dc07d9ccdebdcba5"}
Nov 03 16:34:58.871751 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.871722    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:34:58.935969 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.935935    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume\") pod \"42924e66-4865-4cd3-9771-4e819f4bacb2\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") "
Nov 03 16:34:58.936169 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.936003    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn\" (UniqueName: \"kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn\") pod \"42924e66-4865-4cd3-9771-4e819f4bacb2\" (UID: \"42924e66-4865-4cd3-9771-4e819f4bacb2\") "
Nov 03 16:34:58.936403 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.936366    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume" pod "42924e66-4865-4cd3-9771-4e819f4bacb2" (UID: "42924e66-4865-4cd3-9771-4e819f4bacb2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:34:58.938361 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:58.938325    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn" pod "42924e66-4865-4cd3-9771-4e819f4bacb2" (UID: "42924e66-4865-4cd3-9771-4e819f4bacb2"). InnerVolumeSpecName "kube-api-access-rpttn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:34:59.036838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.036795    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpttn\" (UniqueName: \"kubernetes.io/projected/42924e66-4865-4cd3-9771-4e819f4bacb2-kube-api-access-rpttn\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:59.036838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.036833    4964 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42924e66-4865-4cd3-9771-4e819f4bacb2-extensions-socket-volume\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:34:59.780641 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.780608    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"
Nov 03 16:34:59.780641 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.780622    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9" event={"ID":"42924e66-4865-4cd3-9771-4e819f4bacb2","Type":"ContainerDied","Data":"6c0a7f8251c52af8f89b6557bdbbe92780dd2784d2395fd487e65b2f51b243a0"}
Nov 03 16:34:59.781194 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.780690    4964 scope.go:122] "RemoveContainer" containerID="d15eff275e1337f773f9bd858d99fcc1d40cf2ed37e019d2dc07d9ccdebdcba5"
Nov 03 16:34:59.805103 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.805069    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"]
Nov 03 16:34:59.817365 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:34:59.813143    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bj9c9"]
Nov 03 16:35:00.607312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:00.607272    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42924e66-4865-4cd3-9771-4e819f4bacb2" path="/var/lib/kubelet/pods/42924e66-4865-4cd3-9771-4e819f4bacb2/volumes"
Nov 03 16:35:02.567323 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:35:02.567288    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:35:08.099157 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099117    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"]
Nov 03 16:35:08.099590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099548    4964 cpu_manager.go:414] "containerMap: removing container" podUID="42924e66-4865-4cd3-9771-4e819f4bacb2" containerName="manager"
Nov 03 16:35:08.099590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099560    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42924e66-4865-4cd3-9771-4e819f4bacb2" containerName="manager"
Nov 03 16:35:08.099590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099578    4964 cpu_manager.go:414] "containerMap: removing container" podUID="79eaeeed-263f-4d79-817a-b3d1eefe0682" containerName="revoke-keys"
Nov 03 16:35:08.099590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099583    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79eaeeed-263f-4d79-817a-b3d1eefe0682" containerName="revoke-keys"
Nov 03 16:35:08.099729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099675    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="42924e66-4865-4cd3-9771-4e819f4bacb2" containerName="manager"
Nov 03 16:35:08.099729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.099685    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="79eaeeed-263f-4d79-817a-b3d1eefe0682" containerName="revoke-keys"
Nov 03 16:35:08.104357 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.104328    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.115695 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.115659    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"]
Nov 03 16:35:08.116792 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.116764    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a457934-b922-40b3-a9f9-7d85dd528604-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.116936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.116805    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.116936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.116854    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.116936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.116880    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a457934-b922-40b3-a9f9-7d85dd528604-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.116936 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.116911    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.117122 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.116971    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdt7g\" (UniqueName: \"kubernetes.io/projected/5a457934-b922-40b3-a9f9-7d85dd528604-kube-api-access-sdt7g\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.117122 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.117006    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.117122 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.117033    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.117122 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.117057    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a457934-b922-40b3-a9f9-7d85dd528604-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218083 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218040    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218101    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdt7g\" (UniqueName: \"kubernetes.io/projected/5a457934-b922-40b3-a9f9-7d85dd528604-kube-api-access-sdt7g\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218127    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218146    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218161    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a457934-b922-40b3-a9f9-7d85dd528604-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218220    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a457934-b922-40b3-a9f9-7d85dd528604-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218312 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218250    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218695 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218317    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218695 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218416    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a457934-b922-40b3-a9f9-7d85dd528604-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218695 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218663    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218695 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218678    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218709    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.218855 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.218776    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.219042 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.219026    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a457934-b922-40b3-a9f9-7d85dd528604-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.220543 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.220524    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a457934-b922-40b3-a9f9-7d85dd528604-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.221038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.221020    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a457934-b922-40b3-a9f9-7d85dd528604-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.232017 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.231989    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdt7g\" (UniqueName: \"kubernetes.io/projected/5a457934-b922-40b3-a9f9-7d85dd528604-kube-api-access-sdt7g\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.232139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.231986    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a457934-b922-40b3-a9f9-7d85dd528604-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-zx8xz\" (UID: \"5a457934-b922-40b3-a9f9-7d85dd528604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.424292 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.424259    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:08.710465 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.710437    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"]
Nov 03 16:35:08.712731 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:35:08.712699    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a457934_b922_40b3_a9f9_7d85dd528604.slice/crio-51d7a4dff8b8f5134a194886eff7c871ae05f1ac741100cfdb042bf99d16a0b2 WatchSource:0}: Error finding container 51d7a4dff8b8f5134a194886eff7c871ae05f1ac741100cfdb042bf99d16a0b2: Status 404 returned error can't find the container with id 51d7a4dff8b8f5134a194886eff7c871ae05f1ac741100cfdb042bf99d16a0b2
Nov 03 16:35:08.714937 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.714898    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:35:08.715049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.714983    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:35:08.715049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.715027    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:35:08.830628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.830580    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" event={"ID":"5a457934-b922-40b3-a9f9-7d85dd528604","Type":"ContainerStarted","Data":"b1bb8ec94d00d502a0c169c438775815ea67b798cb10c997b71d03d1bc66b0d9"}
Nov 03 16:35:08.831100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.830657    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" event={"ID":"5a457934-b922-40b3-a9f9-7d85dd528604","Type":"ContainerStarted","Data":"51d7a4dff8b8f5134a194886eff7c871ae05f1ac741100cfdb042bf99d16a0b2"}
Nov 03 16:35:08.856105 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:08.856039    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podStartSLOduration=0.856020739 podStartE2EDuration="856.020739ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:08.8547395 +0000 UTC m=+1970.368121277" watchObservedRunningTime="2026-11-03 16:35:08.856020739 +0000 UTC m=+1970.369402517"
Nov 03 16:35:09.425321 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:09.425280    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:09.426944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:09.426914    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:35:09.427117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:09.426974    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:35:10.428925 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:10.428893    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:10.841827 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:10.841754    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:10.842690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:10.842667    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:35:23.071800 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.071757    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"]
Nov 03 16:35:23.076320 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.076290    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.091902 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.091861    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"]
Nov 03 16:35:23.163675 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163634    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.163861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163687    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.163861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163761    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c298c896-b24a-408c-871f-8c99a8d0b584-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.163861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163801    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.163861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163839    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c298c896-b24a-408c-871f-8c99a8d0b584-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.164008 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163915    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7zv\" (UniqueName: \"kubernetes.io/projected/c298c896-b24a-408c-871f-8c99a8d0b584-kube-api-access-6h7zv\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.164008 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.163950    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.164075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.164027    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c298c896-b24a-408c-871f-8c99a8d0b584-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.164075 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.164048    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265526 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265484    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265563    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265592    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c298c896-b24a-408c-871f-8c99a8d0b584-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265620    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265705 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265674    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c298c896-b24a-408c-871f-8c99a8d0b584-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265704    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7zv\" (UniqueName: \"kubernetes.io/projected/c298c896-b24a-408c-871f-8c99a8d0b584-kube-api-access-6h7zv\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265734    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265793    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c298c896-b24a-408c-871f-8c99a8d0b584-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.265904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.265824    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.266158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.266133    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.266236 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.266168    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.266236 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.266183    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.266340 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.266291    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.266436 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.266412    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c298c896-b24a-408c-871f-8c99a8d0b584-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.268456 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.268425    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c298c896-b24a-408c-871f-8c99a8d0b584-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.268655 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.268632    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c298c896-b24a-408c-871f-8c99a8d0b584-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.279095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.279058    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c298c896-b24a-408c-871f-8c99a8d0b584-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.279228 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.279196    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7zv\" (UniqueName: \"kubernetes.io/projected/c298c896-b24a-408c-871f-8c99a8d0b584-kube-api-access-6h7zv\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8vkzc\" (UID: \"c298c896-b24a-408c-871f-8c99a8d0b584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.391313 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.391227    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:23.673630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.673605    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"]
Nov 03 16:35:23.675650 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:35:23.675626    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc298c896_b24a_408c_871f_8c99a8d0b584.slice/crio-01ffdc1cb4ad38062d637136b687e5170023254a5d2a014add71e7281227c9a6 WatchSource:0}: Error finding container 01ffdc1cb4ad38062d637136b687e5170023254a5d2a014add71e7281227c9a6: Status 404 returned error can't find the container with id 01ffdc1cb4ad38062d637136b687e5170023254a5d2a014add71e7281227c9a6
Nov 03 16:35:23.677815 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.677785    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:35:23.677876 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.677846    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:35:23.677915 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.677872    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:35:23.911459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.911421    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" event={"ID":"c298c896-b24a-408c-871f-8c99a8d0b584","Type":"ContainerStarted","Data":"b04b42d5b87d517fb7bc84e6adf2e1a765d9d98bd8f5f500566db375ade76c1d"}
Nov 03 16:35:23.911653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.911485    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" event={"ID":"c298c896-b24a-408c-871f-8c99a8d0b584","Type":"ContainerStarted","Data":"01ffdc1cb4ad38062d637136b687e5170023254a5d2a014add71e7281227c9a6"}
Nov 03 16:35:23.937376 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:23.937269    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podStartSLOduration=0.937252558 podStartE2EDuration="937.252558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:23.934899349 +0000 UTC m=+1985.448281126" watchObservedRunningTime="2026-11-03 16:35:23.937252558 +0000 UTC m=+1985.450634333"
Nov 03 16:35:24.391989 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:24.391912    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:24.396450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:24.396424    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:24.916459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:24.916425    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:24.917593 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:24.917571    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:35:32.565664 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:35:32.565629    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:35:59.631244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.631162    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"]
Nov 03 16:35:59.635127 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.635102    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.639276 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.639250    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9dbgw\""
Nov 03 16:35:59.647374 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.647344    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"]
Nov 03 16:35:59.716353 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.716310    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/7af4ebcf-7c03-4566-8889-6f454e6bf6bd-kube-api-access-ljx64\") pod \"kuadrant-operator-controller-manager-678d87766f-k7766\" (UID: \"7af4ebcf-7c03-4566-8889-6f454e6bf6bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.716570 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.716479    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af4ebcf-7c03-4566-8889-6f454e6bf6bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k7766\" (UID: \"7af4ebcf-7c03-4566-8889-6f454e6bf6bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.818014 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.817974    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/7af4ebcf-7c03-4566-8889-6f454e6bf6bd-kube-api-access-ljx64\") pod \"kuadrant-operator-controller-manager-678d87766f-k7766\" (UID: \"7af4ebcf-7c03-4566-8889-6f454e6bf6bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.818193 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.818051    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af4ebcf-7c03-4566-8889-6f454e6bf6bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k7766\" (UID: \"7af4ebcf-7c03-4566-8889-6f454e6bf6bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.818482 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.818461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af4ebcf-7c03-4566-8889-6f454e6bf6bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k7766\" (UID: \"7af4ebcf-7c03-4566-8889-6f454e6bf6bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.832741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.832705    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/7af4ebcf-7c03-4566-8889-6f454e6bf6bd-kube-api-access-ljx64\") pod \"kuadrant-operator-controller-manager-678d87766f-k7766\" (UID: \"7af4ebcf-7c03-4566-8889-6f454e6bf6bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:35:59.949984 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:35:59.949949    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:36:00.237741 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:00.237713    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"]
Nov 03 16:36:00.239812 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:36:00.239772    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af4ebcf_7c03_4566_8889_6f454e6bf6bd.slice/crio-fa734fe553c0ff2b9d6dbedc3d68aa73e3496f7b58345bb1968341dca2ce65f4 WatchSource:0}: Error finding container fa734fe553c0ff2b9d6dbedc3d68aa73e3496f7b58345bb1968341dca2ce65f4: Status 404 returned error can't find the container with id fa734fe553c0ff2b9d6dbedc3d68aa73e3496f7b58345bb1968341dca2ce65f4
Nov 03 16:36:01.099634 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:01.099591    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766" event={"ID":"7af4ebcf-7c03-4566-8889-6f454e6bf6bd","Type":"ContainerStarted","Data":"7b9c1e517dea833f1578dd6738452e955a7d5b9cd5b1afecdd78d83cca504189"}
Nov 03 16:36:01.100049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:01.099653    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766" event={"ID":"7af4ebcf-7c03-4566-8889-6f454e6bf6bd","Type":"ContainerStarted","Data":"fa734fe553c0ff2b9d6dbedc3d68aa73e3496f7b58345bb1968341dca2ce65f4"}
Nov 03 16:36:01.100049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:01.099675    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:36:01.127630 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:01.127562    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766" podStartSLOduration=2.1275458560000002 podStartE2EDuration="2.127545856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:01.124640184 +0000 UTC m=+2022.638021960" watchObservedRunningTime="2026-11-03 16:36:01.127545856 +0000 UTC m=+2022.640927638"
Nov 03 16:36:02.567367 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:36:02.567329    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:36:12.107712 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:12.107676    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k7766"
Nov 03 16:36:18.397405 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:18.397343    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:36:26.884555 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:26.884509    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:26.885034 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:26.884591    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:29.290691 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.290655    4964 generic.go:356] "Generic (PLEG): container finished" podID="c298c896-b24a-408c-871f-8c99a8d0b584" containerID="b04b42d5b87d517fb7bc84e6adf2e1a765d9d98bd8f5f500566db375ade76c1d" exitCode=0
Nov 03 16:36:29.291221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.290728    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" event={"ID":"c298c896-b24a-408c-871f-8c99a8d0b584","Type":"ContainerDied","Data":"b04b42d5b87d517fb7bc84e6adf2e1a765d9d98bd8f5f500566db375ade76c1d"}
Nov 03 16:36:29.291221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.291175    4964 scope.go:122] "RemoveContainer" containerID="b04b42d5b87d517fb7bc84e6adf2e1a765d9d98bd8f5f500566db375ade76c1d"
Nov 03 16:36:29.292477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.292443    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:36:29.292601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.292533    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:36:29.292601 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.292580    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:36:29.292720 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.292687    4964 generic.go:356] "Generic (PLEG): container finished" podID="5a457934-b922-40b3-a9f9-7d85dd528604" containerID="b1bb8ec94d00d502a0c169c438775815ea67b798cb10c997b71d03d1bc66b0d9" exitCode=0
Nov 03 16:36:29.292774 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.292740    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" event={"ID":"5a457934-b922-40b3-a9f9-7d85dd528604","Type":"ContainerDied","Data":"b1bb8ec94d00d502a0c169c438775815ea67b798cb10c997b71d03d1bc66b0d9"}
Nov 03 16:36:29.293135 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.293038    4964 scope.go:122] "RemoveContainer" containerID="b1bb8ec94d00d502a0c169c438775815ea67b798cb10c997b71d03d1bc66b0d9"
Nov 03 16:36:29.294044 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.294015    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:36:29.294139 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.294093    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:36:29.294206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:29.294144    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:36:30.299858 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.299823    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" event={"ID":"5a457934-b922-40b3-a9f9-7d85dd528604","Type":"ContainerStarted","Data":"e8ea29b1e2f2168d8ae41346978825f598dfa9efcddc85330c4744c9410d6dab"}
Nov 03 16:36:30.302110 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.302086    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" event={"ID":"c298c896-b24a-408c-871f-8c99a8d0b584","Type":"ContainerStarted","Data":"26dfe43dc09b57c06382bf34bc8ae897a665c37bca1e50e87fc06ef2c9caa4dd"}
Nov 03 16:36:30.391685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.391646    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:36:30.391883 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.391861    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:30.391935 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.391912    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:30.437429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.433763    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:36:30.437429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.434509    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:30.437429 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:30.434565    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:31.391934 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:31.391898    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:31.392538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:31.391960    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:31.434412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:31.433636    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:31.434412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:31.433699    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:32.391907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:32.391866    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:32.392098 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:32.391929    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:32.425412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:32.424824    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:32.425412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:32.424887    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:32.577732 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:36:32.577696    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:36:33.392644 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:33.392608    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:33.393116 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:33.393086    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:33.429410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:33.425845    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:33.429410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:33.425910    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:34.392461 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:34.392421    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:34.392767 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:34.392731    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:34.436410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:34.432824    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:34.436410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:34.432905    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:34.436995 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:34.436956    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:36:35.391912 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:35.391875    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:35.392104 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:35.391931    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:35.424768 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:35.424720    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:35.425206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:35.424791    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:36.392270 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:36.392229    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 16:36:36.392475 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:36.392313    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 16:36:36.424750 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:36.424707    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 16:36:36.424958 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:36.424775    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 16:36:38.392290 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:38.392251    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:38.392760 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:38.392312    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:38.426039 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:38.425987    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:38.426375 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:38.426320    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:38.696353 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:38.696302    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:36:39.438412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:39.437524    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:39.438412 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:39.437610    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:39.480045 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:39.479508    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:39.480045 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:39.479581    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:39.921551 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:39.921505    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:36:40.479590 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:40.479540    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:40.480131 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:40.479607    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:40.527410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:40.526533    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:40.527410 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:40.526644    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:40.842409 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:40.842293    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:36:41.480761 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:41.480716    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:41.481275 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:41.480831    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:41.527355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:41.527302    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:41.527577 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:41.527400    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:42.523729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:42.523674    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:42.524427 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:42.524345    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:42.569699 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:42.569660    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:42.569998 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:42.569959    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz" podUID="5a457934-b922-40b3-a9f9-7d85dd528604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:43.503628 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:43.503569    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:36:43.504921 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:43.504895    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-zx8xz"
Nov 03 16:36:43.545838 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:43.545798    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:36:43.546286 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:43.545895    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc" podUID="c298c896-b24a-408c-871f-8c99a8d0b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:36:43.561961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:43.561926    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:36:43.563122 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:43.563102    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8vkzc"
Nov 03 16:36:53.002802 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:36:53.002766    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:37:01.596280 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:37:01.596242    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:37:02.567197 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:37:02.567150    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:37:16.998632 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:37:16.998591    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:37:21.354497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:37:21.354456    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:37:21.359659 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:37:21.359623    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:37:32.567781 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:37:32.567699    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:37:33.421089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:37:33.421058    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:37:56.113865 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:37:56.113826    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:38:02.566330 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:38:02.566294    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:38:08.272152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:38:08.272116    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:38:19.425322 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:38:19.425278    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:38:32.565312 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:38:32.565272    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:38:36.379005 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:38:36.378962    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:38:51.916961 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:38:51.916924    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:39:02.567112 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:39:02.567032    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:39:04.090112 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:04.090068    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:39:10.816181 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:10.816144    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:39:13.023459 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:13.023411    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"]
Nov 03 16:39:13.031031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:13.030998    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81b3e80-l4js9"]
Nov 03 16:39:14.608685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:14.608648    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed" path="/var/lib/kubelet/pods/3bfb7717-edd5-4eff-b95d-ab52a1e7c1ed/volumes"
Nov 03 16:39:18.911661 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:18.911624    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:39:23.643089 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:23.643036    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:39:32.566474 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:39:32.566431    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:39:39.321548 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:39.321514    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:39:54.786888 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:39:54.786846    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:40:02.566048 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:40:02.566011    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:40:07.099173 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.099133    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"]
Nov 03 16:40:07.102776 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.102754    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.110660 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.110617    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:40:07.110919 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.110896    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\""
Nov 03 16:40:07.111049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.110950    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:40:07.111049 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.110990    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:40:07.113221 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.113193    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"]
Nov 03 16:40:07.141172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.141138    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.141402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.141178    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.141402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.141267    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhz8\" (UniqueName: \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.242200 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.242155    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.242200 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.242199    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.242478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.242277    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhz8\" (UniqueName: \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.243021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.242987    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.244875 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.244854    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.248887 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:40:07.248865    4964 projected.go:196] Error preparing data for projected volume kube-api-access-zxhz8 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 16:40:07.248976 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:40:07.248962    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8 podName:8583500b-bbc1-464e-b0bc-b876b3f4a10b nodeName:}" failed. No retries permitted until 2026-11-03 16:40:07.748946147 +0000 UTC m=+2269.262327924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zxhz8" (UniqueName: "kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8") pod "maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" (UID: "8583500b-bbc1-464e-b0bc-b876b3f4a10b") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 16:40:07.849040 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.848996    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhz8\" (UniqueName: \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:07.869328 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:07.869294    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhz8\" (UniqueName: \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\") pod \"maas-api-revoke-keys-e2e-ait-56d83be3-ppf85\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:08.018953 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.018913    4964 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" secret="" err="secret \"maas-api-cleanup-dockercfg-rqgdz\" not found"
Nov 03 16:40:08.019146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.018975    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:08.301121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.301091    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"]
Nov 03 16:40:08.303332 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:40:08.303301    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8583500b_bbc1_464e_b0bc_b876b3f4a10b.slice/crio-4588de7d645a191f380da0b5db33edb5d1bcb3d5cc4c129b403c2fbe56af5146 WatchSource:0}: Error finding container 4588de7d645a191f380da0b5db33edb5d1bcb3d5cc4c129b403c2fbe56af5146: Status 404 returned error can't find the container with id 4588de7d645a191f380da0b5db33edb5d1bcb3d5cc4c129b403c2fbe56af5146
Nov 03 16:40:08.733964 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.733922    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" event={"ID":"8583500b-bbc1-464e-b0bc-b876b3f4a10b","Type":"ContainerStarted","Data":"cff99c566c612f7be446e0c4bd771b30eeeaaf30224da03d6eb7e5b3b01b12e6"}
Nov 03 16:40:08.734148 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.733985    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" event={"ID":"8583500b-bbc1-464e-b0bc-b876b3f4a10b","Type":"ContainerStarted","Data":"4588de7d645a191f380da0b5db33edb5d1bcb3d5cc4c129b403c2fbe56af5146"}
Nov 03 16:40:08.734206 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.734184    4964 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" secret="" err="secret \"maas-api-cleanup-dockercfg-rqgdz\" not found"
Nov 03 16:40:08.759697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:08.759637    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" podStartSLOduration=1.7596205569999999 podStartE2EDuration="1.759620557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:08.758013035 +0000 UTC m=+2270.271394810" watchObservedRunningTime="2026-11-03 16:40:08.759620557 +0000 UTC m=+2270.273002333"
Nov 03 16:40:09.741514 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:09.741476    4964 generic.go:356] "Generic (PLEG): container finished" podID="8583500b-bbc1-464e-b0bc-b876b3f4a10b" containerID="cff99c566c612f7be446e0c4bd771b30eeeaaf30224da03d6eb7e5b3b01b12e6" exitCode=0
Nov 03 16:40:09.741966 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:09.741572    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" event={"ID":"8583500b-bbc1-464e-b0bc-b876b3f4a10b","Type":"ContainerDied","Data":"cff99c566c612f7be446e0c4bd771b30eeeaaf30224da03d6eb7e5b3b01b12e6"}
Nov 03 16:40:10.888473 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.888443    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:10.975574 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.975540    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca\") pod \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") "
Nov 03 16:40:10.975770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.975653    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca\") pod \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") "
Nov 03 16:40:10.975770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.975681    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\" (UniqueName: \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\") pod \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\" (UID: \"8583500b-bbc1-464e-b0bc-b876b3f4a10b\") "
Nov 03 16:40:10.976074 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.976040    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca" pod "8583500b-bbc1-464e-b0bc-b876b3f4a10b" (UID: "8583500b-bbc1-464e-b0bc-b876b3f4a10b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:40:10.977835 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.977805    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca" pod "8583500b-bbc1-464e-b0bc-b876b3f4a10b" (UID: "8583500b-bbc1-464e-b0bc-b876b3f4a10b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:40:10.977953 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:10.977854    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8" pod "8583500b-bbc1-464e-b0bc-b876b3f4a10b" (UID: "8583500b-bbc1-464e-b0bc-b876b3f4a10b"). InnerVolumeSpecName "kube-api-access-zxhz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:40:11.077136 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:11.077053    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:11.077136 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:11.077086    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxhz8\" (UniqueName: \"kubernetes.io/projected/8583500b-bbc1-464e-b0bc-b876b3f4a10b-kube-api-access-zxhz8\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:11.077136 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:11.077096    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8583500b-bbc1-464e-b0bc-b876b3f4a10b-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:11.754180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:11.754153    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"
Nov 03 16:40:11.754180 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:11.754164    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85" event={"ID":"8583500b-bbc1-464e-b0bc-b876b3f4a10b","Type":"ContainerDied","Data":"4588de7d645a191f380da0b5db33edb5d1bcb3d5cc4c129b403c2fbe56af5146"}
Nov 03 16:40:11.754424 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:11.754207    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4588de7d645a191f380da0b5db33edb5d1bcb3d5cc4c129b403c2fbe56af5146"
Nov 03 16:40:18.002909 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:18.002870    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"]
Nov 03 16:40:18.009853 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:18.009824    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56d83be3-ppf85"]
Nov 03 16:40:18.608585 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:18.608544    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8583500b-bbc1-464e-b0bc-b876b3f4a10b" path="/var/lib/kubelet/pods/8583500b-bbc1-464e-b0bc-b876b3f4a10b/volumes"
Nov 03 16:40:19.295290 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:19.295252    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:40:22.197716 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:22.197679    4964 scope.go:122] "RemoveContainer" containerID="4816ee0c7a2c35ff0728a9737a58426c5831cc74822a660bd5210e4d8bd33d9c"
Nov 03 16:40:22.207067 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:22.207043    4964 scope.go:122] "RemoveContainer" containerID="7f6b8315cb65d36467c9c6f46225606d4221984ff22cb469b79710ea9660e779"
Nov 03 16:40:32.568049 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:40:32.567961    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:40:33.001613 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:40:33.001574    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:41:02.567312 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:41:02.567271    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:41:32.566286 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:41:32.566248    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:42:02.567066 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:42:02.566979    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:42:04.131729 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.131694    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"]
Nov 03 16:42:04.132150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.132117    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8583500b-bbc1-464e-b0bc-b876b3f4a10b" containerName="revoke-keys"
Nov 03 16:42:04.132150 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.132128    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8583500b-bbc1-464e-b0bc-b876b3f4a10b" containerName="revoke-keys"
Nov 03 16:42:04.132244 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.132210    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8583500b-bbc1-464e-b0bc-b876b3f4a10b" containerName="revoke-keys"
Nov 03 16:42:04.135356 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.135335    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.141284 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.141257    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnr82\""
Nov 03 16:42:04.141506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.141486    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:42:04.146557 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.146527    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:42:04.149220 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.149190    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"]
Nov 03 16:42:04.240526 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.240491    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:42:04.285144 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.285106    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.285309 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.285157    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.285309 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.285241    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfg9\" (UniqueName: \"kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.386252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.386149    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.386252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.386219    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.386252 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.386251    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfg9\" (UniqueName: \"kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.386929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.386906    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.388722 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.388702    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.400048 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.400012    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfg9\" (UniqueName: \"kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.449056 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.449014    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:04.732728 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:04.732692    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"]
Nov 03 16:42:04.735323 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:42:04.735290    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5c8375_17e8_472b_9334_066557f0d4b2.slice/crio-8a47bb41c179e274359197aa045558bb952095e0a61fb2578940de89bc48e722 WatchSource:0}: Error finding container 8a47bb41c179e274359197aa045558bb952095e0a61fb2578940de89bc48e722: Status 404 returned error can't find the container with id 8a47bb41c179e274359197aa045558bb952095e0a61fb2578940de89bc48e722
Nov 03 16:42:05.347917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:05.347823    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2" event={"ID":"ac5c8375-17e8-472b-9334-066557f0d4b2","Type":"ContainerStarted","Data":"4fa0fe07c68c83e4c8cf57bc2556124f1fa7b8ad791438f216a1d8d615d79a7d"}
Nov 03 16:42:05.347917 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:05.347881    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2" event={"ID":"ac5c8375-17e8-472b-9334-066557f0d4b2","Type":"ContainerStarted","Data":"8a47bb41c179e274359197aa045558bb952095e0a61fb2578940de89bc48e722"}
Nov 03 16:42:05.368188 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:05.368136    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2" podStartSLOduration=1.368122772 podStartE2EDuration="1.368122772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:05.366291141 +0000 UTC m=+2386.879672919" watchObservedRunningTime="2026-11-03 16:42:05.368122772 +0000 UTC m=+2386.881504547"
Nov 03 16:42:06.354115 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:06.354080    4964 generic.go:356] "Generic (PLEG): container finished" podID="ac5c8375-17e8-472b-9334-066557f0d4b2" containerID="4fa0fe07c68c83e4c8cf57bc2556124f1fa7b8ad791438f216a1d8d615d79a7d" exitCode=0
Nov 03 16:42:06.354552 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:06.354161    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2" event={"ID":"ac5c8375-17e8-472b-9334-066557f0d4b2","Type":"ContainerDied","Data":"4fa0fe07c68c83e4c8cf57bc2556124f1fa7b8ad791438f216a1d8d615d79a7d"}
Nov 03 16:42:07.510498 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.510469    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:07.616669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.616577    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca\") pod \"ac5c8375-17e8-472b-9334-066557f0d4b2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") "
Nov 03 16:42:07.616669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.616638    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9\" (UniqueName: \"kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9\") pod \"ac5c8375-17e8-472b-9334-066557f0d4b2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") "
Nov 03 16:42:07.616916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.616751    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca\") pod \"ac5c8375-17e8-472b-9334-066557f0d4b2\" (UID: \"ac5c8375-17e8-472b-9334-066557f0d4b2\") "
Nov 03 16:42:07.617010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.616981    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca" pod "ac5c8375-17e8-472b-9334-066557f0d4b2" (UID: "ac5c8375-17e8-472b-9334-066557f0d4b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:42:07.617094 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.617075    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:07.618888 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.618856    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9" pod "ac5c8375-17e8-472b-9334-066557f0d4b2" (UID: "ac5c8375-17e8-472b-9334-066557f0d4b2"). InnerVolumeSpecName "kube-api-access-6cfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:42:07.619232 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.619202    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca" pod "ac5c8375-17e8-472b-9334-066557f0d4b2" (UID: "ac5c8375-17e8-472b-9334-066557f0d4b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:42:07.718235 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.718195    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5c8375-17e8-472b-9334-066557f0d4b2-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:07.718476 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:07.718330    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfg9\" (UniqueName: \"kubernetes.io/projected/ac5c8375-17e8-472b-9334-066557f0d4b2-kube-api-access-6cfg9\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:08.366283 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:08.366249    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"
Nov 03 16:42:08.366513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:08.366288    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2" event={"ID":"ac5c8375-17e8-472b-9334-066557f0d4b2","Type":"ContainerDied","Data":"8a47bb41c179e274359197aa045558bb952095e0a61fb2578940de89bc48e722"}
Nov 03 16:42:08.366513 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:08.366350    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a47bb41c179e274359197aa045558bb952095e0a61fb2578940de89bc48e722"
Nov 03 16:42:20.393359 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:20.393315    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"]
Nov 03 16:42:20.396520 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:20.396495    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05bcf4-vb2w2"]
Nov 03 16:42:20.607788 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:20.607752    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5c8375-17e8-472b-9334-066557f0d4b2" path="/var/lib/kubelet/pods/ac5c8375-17e8-472b-9334-066557f0d4b2/volumes"
Nov 03 16:42:21.390869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:21.390838    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:42:21.398290 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:21.398258    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:42:21.782229 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:21.782190    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:42:30.094556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:30.094506    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:42:32.648413 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:42:32.646847    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:42:35.180232 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:35.180191    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:42:39.509708 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.509668    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"]
Nov 03 16:42:39.510146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.510131    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c8375-17e8-472b-9334-066557f0d4b2" containerName="revoke-keys"
Nov 03 16:42:39.510190 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.510147    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c8375-17e8-472b-9334-066557f0d4b2" containerName="revoke-keys"
Nov 03 16:42:39.510260 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.510248    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c8375-17e8-472b-9334-066557f0d4b2" containerName="revoke-keys"
Nov 03 16:42:39.514780 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.514759    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.521683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.521649    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:42:39.521869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.521731    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:42:39.521869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.521818    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:42:39.522021 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.521948    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnr82\""
Nov 03 16:42:39.523010 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.522985    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"]
Nov 03 16:42:39.586756 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.586712    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.586952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.586786    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd2g5\" (UniqueName: \"kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.586952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.586921    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.687801 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.687756    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.687982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.687825    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd2g5\" (UniqueName: \"kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.687982 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.687859    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.688595 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.688564    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.690225 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.690198    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.702082 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.702045    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd2g5\" (UniqueName: \"kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5\") pod \"maas-api-revoke-sub-f03629db-zx4l4\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:39.831179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:39.831092    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:40.115138 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:40.115110    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"]
Nov 03 16:42:40.117918 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:42:40.117890    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ae7ee6_d48a_46fe_9719_df3e05234f5a.slice/crio-b077ae4cf7f960bd3c34ed417369ed5fac0ad12f1c57742afaca5d13ce3f057f WatchSource:0}: Error finding container b077ae4cf7f960bd3c34ed417369ed5fac0ad12f1c57742afaca5d13ce3f057f: Status 404 returned error can't find the container with id b077ae4cf7f960bd3c34ed417369ed5fac0ad12f1c57742afaca5d13ce3f057f
Nov 03 16:42:40.561648 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:40.561606    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4" event={"ID":"c1ae7ee6-d48a-46fe-9719-df3e05234f5a","Type":"ContainerStarted","Data":"5c473ac0da41d6aa206ebe281206aa151720e89311484f3550878a562f47f4bf"}
Nov 03 16:42:40.562172 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:40.561669    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4" event={"ID":"c1ae7ee6-d48a-46fe-9719-df3e05234f5a","Type":"ContainerStarted","Data":"b077ae4cf7f960bd3c34ed417369ed5fac0ad12f1c57742afaca5d13ce3f057f"}
Nov 03 16:42:41.568730 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:41.568692    4964 generic.go:356] "Generic (PLEG): container finished" podID="c1ae7ee6-d48a-46fe-9719-df3e05234f5a" containerID="5c473ac0da41d6aa206ebe281206aa151720e89311484f3550878a562f47f4bf" exitCode=0
Nov 03 16:42:41.569182 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:41.568774    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4" event={"ID":"c1ae7ee6-d48a-46fe-9719-df3e05234f5a","Type":"ContainerDied","Data":"5c473ac0da41d6aa206ebe281206aa151720e89311484f3550878a562f47f4bf"}
Nov 03 16:42:42.708179 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.708151    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:42.818509 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.818474    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca\") pod \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") "
Nov 03 16:42:42.818732 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.818553    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca\") pod \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") "
Nov 03 16:42:42.818732 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.818600    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5\" (UniqueName: \"kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5\") pod \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\" (UID: \"c1ae7ee6-d48a-46fe-9719-df3e05234f5a\") "
Nov 03 16:42:42.818907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.818878    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca" pod "c1ae7ee6-d48a-46fe-9719-df3e05234f5a" (UID: "c1ae7ee6-d48a-46fe-9719-df3e05234f5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:42:42.820735 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.820709    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca" pod "c1ae7ee6-d48a-46fe-9719-df3e05234f5a" (UID: "c1ae7ee6-d48a-46fe-9719-df3e05234f5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:42:42.820839 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.820813    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5" pod "c1ae7ee6-d48a-46fe-9719-df3e05234f5a" (UID: "c1ae7ee6-d48a-46fe-9719-df3e05234f5a"). InnerVolumeSpecName "kube-api-access-bd2g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:42:42.919743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.919696    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:42.919743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.919740    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd2g5\" (UniqueName: \"kubernetes.io/projected/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-kube-api-access-bd2g5\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:42.919743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:42.919755    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ae7ee6-d48a-46fe-9719-df3e05234f5a-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:43.580454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:43.580417    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4" event={"ID":"c1ae7ee6-d48a-46fe-9719-df3e05234f5a","Type":"ContainerDied","Data":"b077ae4cf7f960bd3c34ed417369ed5fac0ad12f1c57742afaca5d13ce3f057f"}
Nov 03 16:42:43.580454 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:43.580450    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zx4l4"
Nov 03 16:42:43.580669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:43.580469    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b077ae4cf7f960bd3c34ed417369ed5fac0ad12f1c57742afaca5d13ce3f057f"
Nov 03 16:42:50.497477 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:50.497438    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:42:55.766247 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.766212    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"]
Nov 03 16:42:55.766704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.766656    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c1ae7ee6-d48a-46fe-9719-df3e05234f5a" containerName="revoke-keys"
Nov 03 16:42:55.766704 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.766667    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ae7ee6-d48a-46fe-9719-df3e05234f5a" containerName="revoke-keys"
Nov 03 16:42:55.766795 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.766732    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ae7ee6-d48a-46fe-9719-df3e05234f5a" containerName="revoke-keys"
Nov 03 16:42:55.770117 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.770102    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.775806 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.775785    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:42:55.777697 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.777670    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnr82\""
Nov 03 16:42:55.781126 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.781101    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"]
Nov 03 16:42:55.829317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.829287    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:42:55.829317 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.829307    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:42:55.889979 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.889937    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.890155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.890089    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zv8\" (UniqueName: \"kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.890155 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.890118    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.990649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.990619    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zv8\" (UniqueName: \"kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.990649 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.990654    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.990852 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.990715    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.991354 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.991332    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:55.993130 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:55.993107    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:56.003441 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:56.003421    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zv8\" (UniqueName: \"kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8\") pod \"maas-api-revoke-sub-0165d7d2-5tk66\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:56.085018 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:56.084936    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:56.368518 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:56.368490    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"]
Nov 03 16:42:56.371398 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:42:56.371356    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2cf5d4_c2e0_440a_84aa_085a5dc0e2ba.slice/crio-0d8d7ec05017dadd32a81aa84266f0d4026092d62a5638c35f32d6dcfccb0cf3 WatchSource:0}: Error finding container 0d8d7ec05017dadd32a81aa84266f0d4026092d62a5638c35f32d6dcfccb0cf3: Status 404 returned error can't find the container with id 0d8d7ec05017dadd32a81aa84266f0d4026092d62a5638c35f32d6dcfccb0cf3
Nov 03 16:42:56.687120 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:56.687077    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66" event={"ID":"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba","Type":"ContainerStarted","Data":"0d8d7ec05017dadd32a81aa84266f0d4026092d62a5638c35f32d6dcfccb0cf3"}
Nov 03 16:42:57.694502 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:57.694468    4964 generic.go:356] "Generic (PLEG): container finished" podID="9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" containerID="80ff3ba9192599773aeeb8e3f2c20730bbf573d5719230a60e1edb3fca62e41e" exitCode=0
Nov 03 16:42:57.694962 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:57.694540    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66" event={"ID":"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba","Type":"ContainerDied","Data":"80ff3ba9192599773aeeb8e3f2c20730bbf573d5719230a60e1edb3fca62e41e"}
Nov 03 16:42:58.840168 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.840138    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:58.915902 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.915858    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca\") pod \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") "
Nov 03 16:42:58.916102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.915933    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8\" (UniqueName: \"kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8\") pod \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") "
Nov 03 16:42:58.916102 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.915956    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca\") pod \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\" (UID: \"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba\") "
Nov 03 16:42:58.916453 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.916419    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca" pod "9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" (UID: "9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:42:58.918195 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.918165    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca" pod "9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" (UID: "9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:42:58.918308 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:58.918213    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8" pod "9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" (UID: "9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba"). InnerVolumeSpecName "kube-api-access-j6zv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:42:59.017085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:59.016982    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:59.017085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:59.017022    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zv8\" (UniqueName: \"kubernetes.io/projected/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-kube-api-access-j6zv8\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:59.017085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:59.017036    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:42:59.706288 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:59.706258    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66"
Nov 03 16:42:59.706500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:59.706295    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5tk66" event={"ID":"9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba","Type":"ContainerDied","Data":"0d8d7ec05017dadd32a81aa84266f0d4026092d62a5638c35f32d6dcfccb0cf3"}
Nov 03 16:42:59.706500 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:42:59.706345    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8d7ec05017dadd32a81aa84266f0d4026092d62a5638c35f32d6dcfccb0cf3"
Nov 03 16:43:02.566150 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:43:02.566116    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:43:06.612578 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.612543    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"]
Nov 03 16:43:06.613213 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.613193    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" containerName="revoke-keys"
Nov 03 16:43:06.613268 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.613216    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" containerName="revoke-keys"
Nov 03 16:43:06.613355 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.613343    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba" containerName="revoke-keys"
Nov 03 16:43:06.617306 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.617282    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.626753 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.626615    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:43:06.629779 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.629731    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnr82\""
Nov 03 16:43:06.630853 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.630832    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"]
Nov 03 16:43:06.686694 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.686655    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjd2p\" (UniqueName: \"kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.686875 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.686795    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.686875 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.686830    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.715152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.715115    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:43:06.715294 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.715214    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:43:06.788040 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.787985    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjd2p\" (UniqueName: \"kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.788265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.788072    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.788265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.788094    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.788828 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.788806    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.790621 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.790598    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.800701 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.800671    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjd2p\" (UniqueName: \"kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p\") pod \"maas-api-revoke-sub-6b19338f-r6grv\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:06.933796 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:06.933761    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:07.225943 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:07.225910    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"]
Nov 03 16:43:07.233663 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:43:07.233531    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d23fcf_3309_4eb7_a34f_2e811a2ab0ed.slice/crio-4f0381eefb7b7e983fe4708098057577f0a7774aaeb28780cf2b16510693cb7f WatchSource:0}: Error finding container 4f0381eefb7b7e983fe4708098057577f0a7774aaeb28780cf2b16510693cb7f: Status 404 returned error can't find the container with id 4f0381eefb7b7e983fe4708098057577f0a7774aaeb28780cf2b16510693cb7f
Nov 03 16:43:07.746765 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:07.746715    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv" event={"ID":"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed","Type":"ContainerStarted","Data":"1e2dd71d226715f300a769d6c3910e7abeacb8ed53a222507acd46258cb07134"}
Nov 03 16:43:07.747192 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:07.746799    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv" event={"ID":"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed","Type":"ContainerStarted","Data":"4f0381eefb7b7e983fe4708098057577f0a7774aaeb28780cf2b16510693cb7f"}
Nov 03 16:43:07.768322 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:07.768198    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv" podStartSLOduration=1.768184303 podStartE2EDuration="1.768184303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:07.76674536 +0000 UTC m=+2449.280127137" watchObservedRunningTime="2026-11-03 16:43:07.768184303 +0000 UTC m=+2449.281566078"
Nov 03 16:43:08.753031 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:08.752994    4964 generic.go:356] "Generic (PLEG): container finished" podID="77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" containerID="1e2dd71d226715f300a769d6c3910e7abeacb8ed53a222507acd46258cb07134" exitCode=0
Nov 03 16:43:08.753517 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:08.753077    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv" event={"ID":"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed","Type":"ContainerDied","Data":"1e2dd71d226715f300a769d6c3910e7abeacb8ed53a222507acd46258cb07134"}
Nov 03 16:43:09.899303 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:09.899270    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:10.021174 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.021086    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p\" (UniqueName: \"kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p\") pod \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") "
Nov 03 16:43:10.021341 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.021185    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca\") pod \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") "
Nov 03 16:43:10.021341 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.021299    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca\") pod \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\" (UID: \"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed\") "
Nov 03 16:43:10.021690 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.021662    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca" pod "77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" (UID: "77d23fcf-3309-4eb7-a34f-2e811a2ab0ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:43:10.023341 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.023313    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca" pod "77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" (UID: "77d23fcf-3309-4eb7-a34f-2e811a2ab0ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:43:10.023341 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.023330    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p" pod "77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" (UID: "77d23fcf-3309-4eb7-a34f-2e811a2ab0ed"). InnerVolumeSpecName "kube-api-access-fjd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:10.123071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.123030    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-openshift-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:10.123071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.123067    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjd2p\" (UniqueName: \"kubernetes.io/projected/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-kube-api-access-fjd2p\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:10.123071 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.123078    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d23fcf-3309-4eb7-a34f-2e811a2ab0ed-maas-api-xks-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:10.764251 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.764217    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv"
Nov 03 16:43:10.764450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.764216    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r6grv" event={"ID":"77d23fcf-3309-4eb7-a34f-2e811a2ab0ed","Type":"ContainerDied","Data":"4f0381eefb7b7e983fe4708098057577f0a7774aaeb28780cf2b16510693cb7f"}
Nov 03 16:43:10.764450 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:10.764346    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0381eefb7b7e983fe4708098057577f0a7774aaeb28780cf2b16510693cb7f"
Nov 03 16:43:17.534281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:17.534241    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jpjhq"]
Nov 03 16:43:29.154669 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:29.154584    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lfxrp_740433eb-df94-4702-beaa-26a4fad6e366/manager/0.log"
Nov 03 16:43:29.362281 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:29.362245    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cljjs_25f59614-ca2f-4f2d-ac69-699714a2f4c5/manager/0.log"
Nov 03 16:43:31.441618 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:31.441586    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-5tk66_9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba/revoke-keys/0.log"
Nov 03 16:43:31.980547 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:31.980514    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-r6grv_77d23fcf-3309-4eb7-a34f-2e811a2ab0ed/revoke-keys/0.log"
Nov 03 16:43:32.336755 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:32.336677    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zx4l4_c1ae7ee6-d48a-46fe-9719-df3e05234f5a/revoke-keys/0.log"
Nov 03 16:43:32.567453 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:43:32.567418    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:43:34.248539 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:34.248501    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9vc9h_8cbdc4a1-7e44-4183-9a1c-a51b22a69623/manager/0.log"
Nov 03 16:43:34.443233 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:34.443203    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-28k8r_4deed5e7-c8e6-4007-9dd0-5b1550dc53b6/manager/0.log"
Nov 03 16:43:34.835703 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:34.835674    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-k7766_7af4ebcf-7c03-4566-8889-6f454e6bf6bd/manager/0.log"
Nov 03 16:43:35.065521 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:35.065482    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jpjhq_003233e0-3e53-410b-90f2-9881e337aa81/limitador/0.log"
Nov 03 16:43:35.260950 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:35.260919    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-r27kw_80ede112-ea2e-4d99-b077-c81612f2429b/manager/0.log"
Nov 03 16:43:36.037085 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:36.037055    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6fwq55k_657478d3-04a7-4e89-9059-14c746334109/istio-proxy/0.log"
Nov 03 16:43:36.951397 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:36.951348    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/1.log"
Nov 03 16:43:38.263121 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:38.263081    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zphl8_e2dc90c6-0535-4ae9-bb93-5c6ee5f68345/istio-proxy/0.log"
Nov 03 16:43:38.444762 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:38.444732    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/1.log"
Nov 03 16:43:38.628243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:38.628163    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr_c5175180-6adf-411c-ab88-b709371d07ba/istio-proxy/0.log"
Nov 03 16:43:39.189739 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:39.189710    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-859db4d5f7-xhncb_2351ca8a-ffba-4733-8413-2ae0b585b489/router/0.log"
Nov 03 16:43:39.629146 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:39.629056    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/0.log"
Nov 03 16:43:40.558262 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:40.558227    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/0.log"
Nov 03 16:43:53.004819 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:53.004783    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lfxrp_740433eb-df94-4702-beaa-26a4fad6e366/manager/0.log"
Nov 03 16:43:53.212212 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:53.212171    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cljjs_25f59614-ca2f-4f2d-ac69-699714a2f4c5/manager/0.log"
Nov 03 16:43:54.978620 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:54.978590    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-5tk66_9f2cf5d4-c2e0-440a-84aa-085a5dc0e2ba/revoke-keys/0.log"
Nov 03 16:43:55.502693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:55.502660    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-r6grv_77d23fcf-3309-4eb7-a34f-2e811a2ab0ed/revoke-keys/0.log"
Nov 03 16:43:55.886212 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:55.886121    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zx4l4_c1ae7ee6-d48a-46fe-9719-df3e05234f5a/revoke-keys/0.log"
Nov 03 16:43:57.430542 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:57.430506    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9vc9h_8cbdc4a1-7e44-4183-9a1c-a51b22a69623/manager/0.log"
Nov 03 16:43:57.614026 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:57.613993    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-28k8r_4deed5e7-c8e6-4007-9dd0-5b1550dc53b6/manager/0.log"
Nov 03 16:43:58.013038 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:58.013007    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-k7766_7af4ebcf-7c03-4566-8889-6f454e6bf6bd/manager/0.log"
Nov 03 16:43:58.224023 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:58.223992    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jpjhq_003233e0-3e53-410b-90f2-9881e337aa81/limitador/0.log"
Nov 03 16:43:58.446152 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:58.446111    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-r27kw_80ede112-ea2e-4d99-b077-c81612f2429b/manager/0.log"
Nov 03 16:43:59.045166 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:59.045134    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6fwq55k_657478d3-04a7-4e89-9059-14c746334109/istio-proxy/0.log"
Nov 03 16:43:59.978129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:43:59.978094    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/1.log"
Nov 03 16:44:01.293944 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:44:01.293912    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zphl8_e2dc90c6-0535-4ae9-bb93-5c6ee5f68345/istio-proxy/0.log"
Nov 03 16:44:01.473869 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:44:01.473835    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/1.log"
Nov 03 16:44:01.658653 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:44:01.658620    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr_c5175180-6adf-411c-ab88-b709371d07ba/istio-proxy/0.log"
Nov 03 16:44:02.191743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:44:02.191710    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-859db4d5f7-xhncb_2351ca8a-ffba-4733-8413-2ae0b585b489/router/0.log"
Nov 03 16:44:02.567351 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:44:02.567261    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:44:32.565847 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:44:32.565811    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:45:00.157334 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.157255    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"]
Nov 03 16:45:00.157809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.157714    4964 cpu_manager.go:414] "containerMap: removing container" podUID="77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" containerName="revoke-keys"
Nov 03 16:45:00.157809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.157728    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" containerName="revoke-keys"
Nov 03 16:45:00.157809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.157804    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d23fcf-3309-4eb7-a34f-2e811a2ab0ed" containerName="revoke-keys"
Nov 03 16:45:00.160809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.160789    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.166584 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.166559    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:45:00.167678 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.167662    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:45:00.167757 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.167676    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnr82\""
Nov 03 16:45:00.189861 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.189822    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"]
Nov 03 16:45:00.327770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.327732    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4fq\" (UniqueName: \"kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq\") pod \"maas-api-key-cleanup-29895405-vtrfd\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.327770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.327774    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-vtrfd\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.428506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.428423    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4fq\" (UniqueName: \"kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq\") pod \"maas-api-key-cleanup-29895405-vtrfd\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.428506 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.428465    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-vtrfd\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.429296 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.429267    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-vtrfd\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.442810 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.442780    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4fq\" (UniqueName: \"kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq\") pod \"maas-api-key-cleanup-29895405-vtrfd\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.472711 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.472680    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:00.748734 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:00.748644    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"]
Nov 03 16:45:00.751981 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:45:00.751950    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8166cc39_72ac_40bb_ae9f_334e99a272d1.slice/crio-4f1053bb99608df6f524a789dc1783b547508c0a2087688535e122679d9ddd5d WatchSource:0}: Error finding container 4f1053bb99608df6f524a789dc1783b547508c0a2087688535e122679d9ddd5d: Status 404 returned error can't find the container with id 4f1053bb99608df6f524a789dc1783b547508c0a2087688535e122679d9ddd5d
Nov 03 16:45:01.379265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:01.379169    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" event={"ID":"8166cc39-72ac-40bb-ae9f-334e99a272d1","Type":"ContainerStarted","Data":"6440e73279b8e0d2b659df4d4b0143604d9b2a9b9a2f3b802aa90b77b800a658"}
Nov 03 16:45:01.379265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:01.379231    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" event={"ID":"8166cc39-72ac-40bb-ae9f-334e99a272d1","Type":"ContainerStarted","Data":"4f1053bb99608df6f524a789dc1783b547508c0a2087688535e122679d9ddd5d"}
Nov 03 16:45:01.397646 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:01.397596    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" podStartSLOduration=1.397578824 podStartE2EDuration="1.397578824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:01.396380234 +0000 UTC m=+2562.909762002" watchObservedRunningTime="2026-11-03 16:45:01.397578824 +0000 UTC m=+2562.910960599"
Nov 03 16:45:02.385422 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:02.385369    4964 generic.go:356] "Generic (PLEG): container finished" podID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerID="6440e73279b8e0d2b659df4d4b0143604d9b2a9b9a2f3b802aa90b77b800a658" exitCode=60
Nov 03 16:45:02.385907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:02.385466    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" event={"ID":"8166cc39-72ac-40bb-ae9f-334e99a272d1","Type":"ContainerDied","Data":"6440e73279b8e0d2b659df4d4b0143604d9b2a9b9a2f3b802aa90b77b800a658"}
Nov 03 16:45:02.385907 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:02.385732    4964 scope.go:122] "RemoveContainer" containerID="6440e73279b8e0d2b659df4d4b0143604d9b2a9b9a2f3b802aa90b77b800a658"
Nov 03 16:45:02.568152 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:45:02.568106    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:45:03.392805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:03.392769    4964 generic.go:356] "Generic (PLEG): container finished" podID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerID="b476bcc011fdd1e790aa3aba8df7c1b5c03cb3ae415c884af0d1fd5706b766e6" exitCode=60
Nov 03 16:45:03.393238 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:03.392853    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" event={"ID":"8166cc39-72ac-40bb-ae9f-334e99a272d1","Type":"ContainerDied","Data":"b476bcc011fdd1e790aa3aba8df7c1b5c03cb3ae415c884af0d1fd5706b766e6"}
Nov 03 16:45:03.393238 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:03.392916    4964 scope.go:122] "RemoveContainer" containerID="6440e73279b8e0d2b659df4d4b0143604d9b2a9b9a2f3b802aa90b77b800a658"
Nov 03 16:45:03.393238 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:03.393176    4964 scope.go:122] "RemoveContainer" containerID="b476bcc011fdd1e790aa3aba8df7c1b5c03cb3ae415c884af0d1fd5706b766e6"
Nov 03 16:45:03.393490 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:45:03.393472    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-vtrfd_odh-ai-gateway-infra(8166cc39-72ac-40bb-ae9f-334e99a272d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1"
Nov 03 16:45:04.399761 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:04.399728    4964 scope.go:122] "RemoveContainer" containerID="b476bcc011fdd1e790aa3aba8df7c1b5c03cb3ae415c884af0d1fd5706b766e6"
Nov 03 16:45:04.400183 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:45:04.399947    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-vtrfd_odh-ai-gateway-infra(8166cc39-72ac-40bb-ae9f-334e99a272d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1"
Nov 03 16:45:16.601836 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:16.601804    4964 scope.go:122] "RemoveContainer" containerID="b476bcc011fdd1e790aa3aba8df7c1b5c03cb3ae415c884af0d1fd5706b766e6"
Nov 03 16:45:17.465872 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:17.465839    4964 generic.go:356] "Generic (PLEG): container finished" podID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerID="551d4c0e98d104a7ecc85e9be303cdf6ae7188b35520573594712143000255a0" exitCode=60
Nov 03 16:45:17.466036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:17.465911    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" event={"ID":"8166cc39-72ac-40bb-ae9f-334e99a272d1","Type":"ContainerDied","Data":"551d4c0e98d104a7ecc85e9be303cdf6ae7188b35520573594712143000255a0"}
Nov 03 16:45:17.466036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:17.466001    4964 scope.go:122] "RemoveContainer" containerID="b476bcc011fdd1e790aa3aba8df7c1b5c03cb3ae415c884af0d1fd5706b766e6"
Nov 03 16:45:17.466347 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:17.466331    4964 scope.go:122] "RemoveContainer" containerID="551d4c0e98d104a7ecc85e9be303cdf6ae7188b35520573594712143000255a0"
Nov 03 16:45:17.466605 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:45:17.466585    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-vtrfd_odh-ai-gateway-infra(8166cc39-72ac-40bb-ae9f-334e99a272d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1"
Nov 03 16:45:18.496748 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.496711    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"]
Nov 03 16:45:18.633811 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.633784    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:18.701255 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.701218    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq\" (UniqueName: \"kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq\") pod \"8166cc39-72ac-40bb-ae9f-334e99a272d1\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") "
Nov 03 16:45:18.701497 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.701278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca\") pod \"8166cc39-72ac-40bb-ae9f-334e99a272d1\" (UID: \"8166cc39-72ac-40bb-ae9f-334e99a272d1\") "
Nov 03 16:45:18.701737 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.701699    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca" pod "8166cc39-72ac-40bb-ae9f-334e99a272d1" (UID: "8166cc39-72ac-40bb-ae9f-334e99a272d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:45:18.701914 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.701888    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8166cc39-72ac-40bb-ae9f-334e99a272d1-maas-api-service-ca\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:18.703773 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.703752    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq" pod "8166cc39-72ac-40bb-ae9f-334e99a272d1" (UID: "8166cc39-72ac-40bb-ae9f-334e99a272d1"). InnerVolumeSpecName "kube-api-access-rt4fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:45:18.803265 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:18.803185    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4fq\" (UniqueName: \"kubernetes.io/projected/8166cc39-72ac-40bb-ae9f-334e99a272d1-kube-api-access-rt4fq\") on node \"ip-10-0-133-129.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:19.479243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:19.479214    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"
Nov 03 16:45:19.479443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:19.479214    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd" event={"ID":"8166cc39-72ac-40bb-ae9f-334e99a272d1","Type":"ContainerDied","Data":"4f1053bb99608df6f524a789dc1783b547508c0a2087688535e122679d9ddd5d"}
Nov 03 16:45:19.479443 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:19.479341    4964 scope.go:122] "RemoveContainer" containerID="551d4c0e98d104a7ecc85e9be303cdf6ae7188b35520573594712143000255a0"
Nov 03 16:45:19.503396 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:19.503360    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"]
Nov 03 16:45:19.512478 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:19.512452    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-vtrfd"]
Nov 03 16:45:20.607913 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:20.607873    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" path="/var/lib/kubelet/pods/8166cc39-72ac-40bb-ae9f-334e99a272d1/volumes"
Nov 03 16:45:32.566270 ip-10-0-133-129 kubenswrapper[4964]: E1103 16:45:32.566234    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bcvq5" containerName="extractor"
Nov 03 16:45:33.655173 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:33.655142    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rtlt9_d3387caf-55fb-4a86-9b89-f38c8fbe1915/global-pull-secret-syncer/0.log"
Nov 03 16:45:33.771932 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:33.771903    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vsbds_dfe05141-01c7-4446-a5b5-e61abe65dfa2/connection-checker/0.log"
Nov 03 16:45:33.802541 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:33.802515    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wlfxh_fc94f4e0-d6f9-4b11-8495-36cdf02aa8a2/connection-checker/0.log"
Nov 03 16:45:33.831367 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:33.831343    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xw2n8_315b5fc8-0084-4395-96a5-8ae4e210b974/connection-checker/0.log"
Nov 03 16:45:33.919133 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:33.919055    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r59q4_f3bb7151-48a1-4b4d-9784-31da848ac4f0/konnectivity-agent/0.log"
Nov 03 16:45:34.028904 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:34.028875    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-129.us-east-2.compute.internal_19bdd4315736b0c6d37ff28999a14800/haproxy/0.log"
Nov 03 16:45:41.256929 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:41.256881    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9vc9h_8cbdc4a1-7e44-4183-9a1c-a51b22a69623/manager/0.log"
Nov 03 16:45:41.293471 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:41.293443    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-28k8r_4deed5e7-c8e6-4007-9dd0-5b1550dc53b6/manager/0.log"
Nov 03 16:45:41.407770 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:41.407738    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-k7766_7af4ebcf-7c03-4566-8889-6f454e6bf6bd/manager/0.log"
Nov 03 16:45:41.453538 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:41.453499    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jpjhq_003233e0-3e53-410b-90f2-9881e337aa81/limitador/0.log"
Nov 03 16:45:41.517058 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:41.516987    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-r27kw_80ede112-ea2e-4d99-b077-c81612f2429b/manager/0.log"
Nov 03 16:45:43.066123 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.066088    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-7ql5t_d7a6ba52-20d8-40fe-9723-54418819a3ab/cluster-monitoring-operator/0.log"
Nov 03 16:45:43.476444 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.476417    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t4w2k_70b0c8dc-d244-46ea-a229-389133d58c51/node-exporter/0.log"
Nov 03 16:45:43.502868 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.502842    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t4w2k_70b0c8dc-d244-46ea-a229-389133d58c51/kube-rbac-proxy/0.log"
Nov 03 16:45:43.529683 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.529659    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t4w2k_70b0c8dc-d244-46ea-a229-389133d58c51/init-textfile/0.log"
Nov 03 16:45:43.675654 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.675621    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"]
Nov 03 16:45:43.676065 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676050    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676120 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676067    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676120 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676078    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676120 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676083    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676120 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676099    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676120 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676103    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676279 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676194    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676279 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676201    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.676279 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.676208    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8166cc39-72ac-40bb-ae9f-334e99a272d1" containerName="cleanup"
Nov 03 16:45:43.680871 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.680848    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.690763 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.690738    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"]
Nov 03 16:45:43.777036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.776949    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2xlg\"/\"kube-root-ca.crt\""
Nov 03 16:45:43.777036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.776966    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v2xlg\"/\"default-dockercfg-qmzbm\""
Nov 03 16:45:43.777036 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.776949    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2xlg\"/\"openshift-service-ca.crt\""
Nov 03 16:45:43.844678 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.844646    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-lib-modules\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.844863 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.844687    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8blx\" (UniqueName: \"kubernetes.io/projected/7f9ba79d-3a5d-462c-9577-ba521603f150-kube-api-access-z8blx\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.844863 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.844710    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-proc\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.844863 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.844855    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-podres\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.845022 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.844901    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-sys\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.927743 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.927715    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bzxr6_f35af85a-5764-4e7d-a673-3e7fb1883d0d/prometheus-operator/0.log"
Nov 03 16:45:43.945805 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945778    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-lib-modules\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.945939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945815    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8blx\" (UniqueName: \"kubernetes.io/projected/7f9ba79d-3a5d-462c-9577-ba521603f150-kube-api-access-z8blx\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.945939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945840    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-proc\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.945939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945894    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-podres\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.945939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945910    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-lib-modules\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.945939 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945928    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-sys\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.946129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945974    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-proc\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.946129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.945979    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-sys\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.946129 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.946005    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f9ba79d-3a5d-462c-9577-ba521603f150-podres\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.957818 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.957790    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8blx\" (UniqueName: \"kubernetes.io/projected/7f9ba79d-3a5d-462c-9577-ba521603f150-kube-api-access-z8blx\") pod \"perf-node-gather-daemonset-bvlhj\" (UID: \"7f9ba79d-3a5d-462c-9577-ba521603f150\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:43.963979 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.963941    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bzxr6_f35af85a-5764-4e7d-a673-3e7fb1883d0d/kube-rbac-proxy/0.log"
Nov 03 16:45:43.992813 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.992791    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7dmkl_3c2be6f7-3fae-4cb9-9615-3a3d15140cec/prometheus-operator-admission-webhook/0.log"
Nov 03 16:45:43.994916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:43.994898    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:44.274949 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:44.274926    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"]
Nov 03 16:45:44.277733 ip-10-0-133-129 kubenswrapper[4964]: W1103 16:45:44.277709    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f9ba79d_3a5d_462c_9577_ba521603f150.slice/crio-c44c32c80ef80a199d726387c7433fd9c0015d97f69493d2b8374a1e4dc0ad73 WatchSource:0}: Error finding container c44c32c80ef80a199d726387c7433fd9c0015d97f69493d2b8374a1e4dc0ad73: Status 404 returned error can't find the container with id c44c32c80ef80a199d726387c7433fd9c0015d97f69493d2b8374a1e4dc0ad73
Nov 03 16:45:44.617693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:44.617606    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj" event={"ID":"7f9ba79d-3a5d-462c-9577-ba521603f150","Type":"ContainerStarted","Data":"f51e98bf453393c8078fea4a9656cf4cd7194a449673ceb631a1d5f4d40426c3"}
Nov 03 16:45:44.617693 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:44.617669    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj" event={"ID":"7f9ba79d-3a5d-462c-9577-ba521603f150","Type":"ContainerStarted","Data":"c44c32c80ef80a199d726387c7433fd9c0015d97f69493d2b8374a1e4dc0ad73"}
Nov 03 16:45:44.617885 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:44.617822    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:44.637943 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:44.637894    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj" podStartSLOduration=1.6378796599999998 podStartE2EDuration="1.63787966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:44.637100477 +0000 UTC m=+2606.150482267" watchObservedRunningTime="2026-11-03 16:45:44.63787966 +0000 UTC m=+2606.151261432"
Nov 03 16:45:45.394243 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:45.394210    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-8bw8n_6ab65c33-a23d-48f3-ba16-212af80c62cb/networking-console-plugin/0.log"
Nov 03 16:45:46.089624 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:46.089597    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/2.log"
Nov 03 16:45:46.094044 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:46.094020    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4ftv2_1b592ced-7c65-40a8-94d7-cad2c29fe081/console-operator/3.log"
Nov 03 16:45:46.629556 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:46.629525    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-xrkn4_64dd3f55-3e92-4fb9-a122-694296b32596/download-server/0.log"
Nov 03 16:45:47.235924 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:47.235896    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-qfwp4_d327c29c-36d3-4d01-8cab-bf10a76dbf48/volume-data-source-validator/0.log"
Nov 03 16:45:48.471916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:48.471892    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4wvw_66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/dns/0.log"
Nov 03 16:45:48.500609 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:48.500586    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4wvw_66c8687d-b9da-46b2-85ce-52cd8f6ac1dd/kube-rbac-proxy/0.log"
Nov 03 16:45:48.671353 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:48.671325    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mlccd_6a30fac5-ac40-4179-9628-fa3203ddb77b/dns-node-resolver/0.log"
Nov 03 16:45:49.305959 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:49.305925    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-556f6f5cbf-277c2_ab643762-ad17-432d-9190-b1c6fd2f6a34/registry/0.log"
Nov 03 16:45:49.329306 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:49.329275    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5wvsc_66a8e707-3e0b-42f2-988f-517d834a869c/node-ca/0.log"
Nov 03 16:45:50.537945 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:50.537917    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6fwq55k_657478d3-04a7-4e89-9059-14c746334109/istio-proxy/0.log"
Nov 03 16:45:50.633685 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:50.633659    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-bvlhj"
Nov 03 16:45:52.927656 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:52.927625    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/1.log"
Nov 03 16:45:52.928158 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:52.927735    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/0.log"
Nov 03 16:45:53.187176 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:53.187101    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zphl8_e2dc90c6-0535-4ae9-bb93-5c6ee5f68345/istio-proxy/0.log"
Nov 03 16:45:53.220887 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:53.220861    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/1.log"
Nov 03 16:45:53.221095 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:53.221079    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/0.log"
Nov 03 16:45:53.252318 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:53.252290    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr_c5175180-6adf-411c-ab88-b709371d07ba/istio-proxy/0.log"
Nov 03 16:45:53.377474 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:53.377445    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-859db4d5f7-xhncb_2351ca8a-ffba-4733-8413-2ae0b585b489/router/0.log"
Nov 03 16:45:54.178001 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:54.177973    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6fwq55k_657478d3-04a7-4e89-9059-14c746334109/istio-proxy/0.log"
Nov 03 16:45:54.898525 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:54.898496    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8qq6k_e3dbcf1c-4684-46e8-b806-1df38410a5c9/serve-healthcheck-canary/0.log"
Nov 03 16:45:55.468256 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:55.468216    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-vrzhj_bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/insights-operator/0.log"
Nov 03 16:45:55.469583 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:55.469561    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-vrzhj_bb4f4939-d15b-4b06-bc6e-27ca8b2039ec/insights-operator/1.log"
Nov 03 16:45:55.595499 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:55.595474    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bcvq5_4beaa779-098d-4007-88f8-25796f1758f0/kube-rbac-proxy/0.log"
Nov 03 16:45:55.626402 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:55.626363    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bcvq5_4beaa779-098d-4007-88f8-25796f1758f0/exporter/0.log"
Nov 03 16:45:55.657901 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:55.657877    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bcvq5_4beaa779-098d-4007-88f8-25796f1758f0/extractor/0.log"
Nov 03 16:45:55.795721 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:55.795646    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-49czb-t8nxv_4cb7e596-9c64-4bcc-8841-45d1fd5e0485/insights-gathering/0.log"
Nov 03 16:45:56.609916 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.609860    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/1.log"
Nov 03 16:45:56.610330 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.609889    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8vkzc_c298c896-b24a-408c-871f-8c99a8d0b584/istio-proxy/0.log"
Nov 03 16:45:56.730952 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.730925    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zphl8_e2dc90c6-0535-4ae9-bb93-5c6ee5f68345/istio-proxy/0.log"
Nov 03 16:45:56.750381 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.750330    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/1.log"
Nov 03 16:45:56.750644 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.750625    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-zx8xz_5a457934-b922-40b3-a9f9-7d85dd528604/istio-proxy/0.log"
Nov 03 16:45:56.767182 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.767155    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-xbhrr_c5175180-6adf-411c-ab88-b709371d07ba/istio-proxy/0.log"
Nov 03 16:45:56.850100 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:56.850074    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-859db4d5f7-xhncb_2351ca8a-ffba-4733-8413-2ae0b585b489/router/0.log"
Nov 03 16:45:58.343299 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:58.343272    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-d59cb86d5-6q2wx_7c4b613c-7266-4dc0-8bc6-f76cc75947c7/manager/0.log"
Nov 03 16:45:58.374216 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:58.374191    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-d59cb86d5-6q2wx_7c4b613c-7266-4dc0-8bc6-f76cc75947c7/copy-manifests/0.log"
Nov 03 16:45:58.855809 ip-10-0-133-129 kubenswrapper[4964]: I1103 16:45:58.855780    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6fwq55k_657478d3-04a7-4e89-9059-14c746334109/istio-proxy/0.log"