Oct 30 20:03:14.399269 ip-10-0-141-189 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:03:14.721625 ip-10-0-141-189 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:03:14.721625 ip-10-0-141-189 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:03:14.721625 ip-10-0-141-189 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:03:14.721625 ip-10-0-141-189 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:03:14.726406 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726331    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.726406 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726406    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.726406 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726410    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726414    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726417    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726420    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726423    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726427    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726430    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726435    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726440    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726443    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726447    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726450    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726453    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726456    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726458    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726461    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726464    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726467    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726470    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.726498 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726472    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726478    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726481    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726484    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726486    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726489    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726492    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726494    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726497    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726500    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726502    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726505    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726508    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726511    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726513    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726516    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726519    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726521    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726524    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726526    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.726940 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726529    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726532    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726535    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726537    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726540    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726543    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726546    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726548    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726551    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726553    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726556    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726559    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726562    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726565    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726568    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726570    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726573    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726576    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726578    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.727426 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726581    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726584    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726586    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726589    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726592    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726594    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726597    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726599    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726602    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726605    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726607    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726609    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726612    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726615    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726617    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726620    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726622    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726624    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726627    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726630    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.727909 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726633    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726635    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726638    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726640    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726646    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726649    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726652    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726655    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726658    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726660    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726664    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726667    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726669    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726672    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726674    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726677    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726686    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726690    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726693    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.728387 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726696    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726699    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726702    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726704    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726708    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726711    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.726714    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727508    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727515    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727517    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727520    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727523    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727526    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727535    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727538    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727541    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727546    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727552    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727555    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.728830 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727558    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727561    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727564    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727567    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727570    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727572    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727575    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727577    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727580    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727582    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727585    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727588    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727590    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727593    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727595    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727598    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727600    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727603    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727606    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727609    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.729297 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727611    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727614    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727617    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727619    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727622    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727624    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727627    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727637    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727640    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727643    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727646    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727648    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727651    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727653    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727656    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727658    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727660    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727663    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727668    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.729775 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727671    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727673    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727676    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727678    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727681    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727684    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727686    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727689    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727691    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727694    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727697    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727700    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727702    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727705    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727707    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727710    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727712    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727715    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727718    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.730260 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727720    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727723    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727726    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727729    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727732    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727734    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727737    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727740    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727742    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727746    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727750    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727754    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727757    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727760    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727762    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727765    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727767    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727770    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727772    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727775    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.730755 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727778    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727780    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727784    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727786    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727789    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727791    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727794    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727796    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727799    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727801    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727804    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727807    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727809    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727813    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.727819    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728324    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728329    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728332    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728335    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.731249 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728337    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728340    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728343    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728346    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728348    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728351    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728354    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728356    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728359    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728362    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728364    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728367    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728370    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728372    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728376    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728379    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728382    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728384    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728387    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728389    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.731698 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728392    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728395    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728397    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728403    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728405    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728408    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728411    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728414    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728417    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728419    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728422    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728425    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728429    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728433    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728436    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728440    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728443    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728446    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728449    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.732203 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728452    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728455    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728458    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728460    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728463    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728465    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728468    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728470    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728473    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728476    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728479    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728481    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728484    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728487    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728489    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728491    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728496    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728499    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728501    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728504    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.732647 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728507    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728510    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728512    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728515    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728517    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728520    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728522    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728525    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728528    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728530    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728533    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728535    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728538    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728541    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728543    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728546    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728548    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728551    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728554    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728557    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.733135 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728560    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728562    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728565    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728567    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728570    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728574    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728577    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728580    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728585    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728588    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728591    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728593    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728596    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728602    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728604    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728607    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728609    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728613    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.733600 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728615    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728618    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728620    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.728623    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729552    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729562    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729569    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729573    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729580    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729583    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729588    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729592    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729595    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729599    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729603    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729606    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729609    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729613    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729616    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729619    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729621    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729626    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729631    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:03:14.734055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729634    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729637    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729640    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729644    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729648    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729652    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729655    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729658    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729662    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729665    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729668    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729671    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729674    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729678    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729681    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729684    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729687    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729690    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729693    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729697    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729700    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729703    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729709    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729712    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:03:14.734608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729716    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729719    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729722    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729726    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729729    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729732    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729737    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729740    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729742    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729746    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729749    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729753    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729756    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729759    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729763    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729766    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729769    2577 flags.go:64] FLAG: --help="false"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729772    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729775    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729779    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729782    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729785    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729789    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:03:14.735229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729792    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729795    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729798    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729801    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729804    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729807    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729810    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729814    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729818    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729821    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729824    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729827    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729830    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729833    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729836    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729843    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729846    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729849    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729851    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729855    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729858    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729862    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729865    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729869    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729873    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:03:14.735825 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729878    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729881    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729884    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729887    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729890    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729894    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729897    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729900    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729906    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729909    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729912    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729915    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729918    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729921    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729926    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729929    2577 flags.go:64] FLAG: --port="10250"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729932    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729935    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07c49397d0d488fd1"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729939    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729941    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729944    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729947    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729953    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729956    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729959    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729962    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:03:14.736534 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729966    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729969    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729972    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729975    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729978    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729981    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729985    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729988    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729991    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729994    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.729997    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730000    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730014    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730017    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730020    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730024    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730027    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730030    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730033    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730036    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730042    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730045    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730048    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730053    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730056    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:03:14.737171 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730059    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730061    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730064    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730069    2577 flags.go:64] FLAG: --v="2"
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730073    2577 flags.go:64] FLAG: --version="false"
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730077    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730081    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730084    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730228    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730232    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730236    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730238    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730241    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730244    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730247    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730250    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730253    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730259    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730262    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730265    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730268    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.737757 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730271    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730273    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730276    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730279    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730282    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730285    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730288    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730290    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730293    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730296    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730300    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730304    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730307    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730317    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730322    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730325    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730328    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730330    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730333    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.738290 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730335    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730338    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730341    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730343    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730346    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730348    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730352    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730354    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730357    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730360    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730362    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730365    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730367    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730370    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730372    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730375    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730377    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730380    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730383    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730385    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.738741 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730388    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730390    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730393    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730396    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730399    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730404    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730407    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730409    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730412    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730414    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730417    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730419    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730422    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730424    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730427    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730430    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730432    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730435    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730438    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.739252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730441    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730443    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730447    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730451    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730454    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730457    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730460    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730462    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730465    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730468    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730471    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730473    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730475    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730478    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730481    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730483    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730486    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730488    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730491    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.739760 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730493    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730496    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730498    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730501    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730504    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730506    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730509    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730511    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730514    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730517    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730519    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730522    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730524    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730527    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.730552    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730590    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.740332 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730594    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730598    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730601    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730603    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730606    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730609    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730611    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730614    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730616    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730619    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730621    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730624    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730626    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730629    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730632    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730634    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730637    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730640    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730642    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.740746 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730645    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730648    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730650    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730657    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730660    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730662    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730665    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730668    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730670    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730673    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730676    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730679    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730681    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730684    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730687    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730689    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730692    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730694    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730697    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.741274 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730699    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730702    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730705    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730707    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730711    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730714    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730716    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730719    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730721    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730724    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730727    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730729    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730732    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730735    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730739    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730743    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730747    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730751    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730753    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.741752 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730756    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730759    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730762    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730764    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730767    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730771    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730773    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730776    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730779    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730781    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730784    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730787    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730789    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730792    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730795    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730797    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730800    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730804    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730806    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730809    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.742220 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730814    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730817    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730819    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730822    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730825    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730827    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730830    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730833    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730837    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730839    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730842    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730844    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730847    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730849    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730852    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730855    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730857    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730860    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.742700 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730863    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730865    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730868    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730870    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730873    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730875    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730878    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.730883    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.738042    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.738057    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.738073    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.738078    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738134    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738138    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738142    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738145    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738148    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738151    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738153    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.743163 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738156    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738159    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738162    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738164    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738167    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738170    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738172    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738175    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738177    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738180    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738183    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738186    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738188    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738191    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738194    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738197    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738200    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738202    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738205    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738208    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.743608 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738211    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738214    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738217    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738220    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738222    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738225    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738228    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738230    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738233    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738235    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738238    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738241    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738243    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738246    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738248    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738252    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738254    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738257    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738259    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.744097 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738262    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738265    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738268    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738270    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738273    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738275    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738278    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738280    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738283    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738286    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738288    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738291    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738294    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738299    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738302    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738306    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738309    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738318    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738321    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738324    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.744542 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738327    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738330    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738333    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738336    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738338    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738341    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738344    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738347    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738349    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738352    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738354    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738357    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738359    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738362    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738364    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738367    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738369    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738374    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.745081 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738377    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738381    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738384    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738388    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738390    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738393    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738396    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738399    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738402    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738404    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738407    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738410    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738412    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738415    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738417    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738420    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738422    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738425    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738428    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.745524 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738430    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738433    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.738459    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738596    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738601    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738604    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738607    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738610    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738613    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738615    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738618    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738621    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738623    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738626    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738628    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738631    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.745976 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738633    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738637    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738639    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738642    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738645    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738648    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738651    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738653    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738656    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738658    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738661    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738664    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738666    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738669    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738671    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738674    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738676    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738679    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738681    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738684    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.746368 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738686    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738688    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738691    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738693    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738696    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738699    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738701    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738703    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738706    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738708    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738711    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738713    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738716    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738718    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738721    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738724    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738726    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738729    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738732    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738734    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.746835 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738737    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738739    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738742    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738745    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738748    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738751    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738755    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738759    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738763    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738766    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738768    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738771    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738773    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738776    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738778    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738782    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738785    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738788    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.747308 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738791    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738793    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738796    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738798    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738801    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738803    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738806    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738808    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738811    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738814    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738817    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738820    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738822    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738825    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738827    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738830    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738833    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738835    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738838    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.747721 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738840    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738843    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738845    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738848    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738851    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738853    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738856    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738861    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738864    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738866    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738869    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738871    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738874    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738876    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738879    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.748306 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.738900    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738929    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738932    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738936    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738939    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738941    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738944    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738947    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738953    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738955    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738958    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738960    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738963    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738966    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738968    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738971    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738974    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738977    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738979    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738982    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:03:14.748682 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738984    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738987    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738990    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738992    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738995    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.738997    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739000    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739016    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739020    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739022    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739025    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739028    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739031    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739033    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739036    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739038    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739041    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739043    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739046    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739048    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:03:14.749160 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739054    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739057    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739059    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739062    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739064    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739067    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739070    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739072    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739075    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739078    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739080    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739083    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739087    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739091    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739095    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739099    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739102    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739105    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739107    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:03:14.749614 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739110    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739113    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739118    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739121    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739124    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739127    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739129    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739132    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739135    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739137    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739140    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739142    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739145    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739150    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739154    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739156    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739159    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739162    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739164    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739167    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:03:14.750155 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739170    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739173    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739175    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739178    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739180    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739183    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739186    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739188    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739191    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739193    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739196    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739198    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739201    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739204    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739206    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739209    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739211    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739214    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739216    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739219    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:03:14.750678 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739221    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739224    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739226    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739229    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739231    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739237    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739239    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:14.739273    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.741106    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.744322    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.745350    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.745443    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.746078    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.748213    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:03:14.751247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.749743    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:03:14.759779 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.759765    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:03:14.763609 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.763596    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 20:03:14.764497 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.764472    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:03:14.770348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.770329    2577 fs.go:135] Filesystem UUIDs: map[3da0bb67-0e66-4cba-9541-19664a699fff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ece864de-a339-4c3e-8361-383eaa27df37:/dev/nvme0n1p3]
Oct 30 20:03:14.770399 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.770348    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:03:14.775773 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.775669    2577 manager.go:217] Machine: {Timestamp:2026-10-30 20:03:14.774348952 +0000 UTC m=+0.281771400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079772 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f75405e2b77da717ea74ea87382e SystemUUID:ec22f754-05e2-b77d-a717-ea74ea87382e BootID:32b3e0c4-e7b3-40e2-852e-8071218f6f57 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:ac:b7:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:ac:b7:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:e7:42:45:f7:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:03:14.775842 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.775775    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:03:14.775883 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.775872    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:03:14.777141 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.777122    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:03:14.777284 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.777144    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-189.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:03:14.777986 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.777976    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:03:14.778026 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.777987    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:03:14.778059 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.778028    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:03:14.778059 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.778044    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:03:14.779281 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.779271    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:03:14.783927 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.783916    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:03:14.784093 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.784083    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:03:14.784334 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.784324    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:03:14.784381 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.784372    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:03:14.784409 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.784388    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:03:14.784409 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.784400    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:03:14.784502 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.784491    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:03:14.787474 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.787418    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:03:14.790195 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.790178    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:03:14.790708 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.790688    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:03:14.796823 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.796808    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:03:14.797713 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797702    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:03:14.797767 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797731    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:03:14.797876 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797865    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:03:14.797910 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797883    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:03:14.797910 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797890    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:03:14.797910 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797896    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:03:14.797910 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797901    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:03:14.797910 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.797907    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:03:14.798607 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.798597    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:03:14.798648 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.798608    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:03:14.798648 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.798615    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:03:14.798648 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.798622    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:03:14.798648 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.798630    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:03:14.798648 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.798639    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:03:14.800130 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.800121    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:03:14.800130 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.800131    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:03:14.804143 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.804130    2577 server.go:1263] "Started kubelet"
Oct 30 20:03:14.804223 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.804196    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:03:14.804423 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.804253    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:03:14.804423 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.804324    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:03:14.804911 ip-10-0-141-189 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:03:14.805892 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.805874    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:03:14.806214 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.806198    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-189.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:03:14.806300 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.806227    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-189.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 20:03:14.806300 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.806237    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 20:03:14.806442 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.806430    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:03:14.812631 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.812611    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:03:14.814809 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.814791    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:03:14.816125 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.816107    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:03:14.816224 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.816205    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:14.816876 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.816858    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:03:14.816876 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.816874    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:03:14.816995 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.816857    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:03:14.816995 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.816948    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:03:14.816995 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.816956    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:03:14.817511 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817497    2577 factory.go:55] Registering systemd factory
Oct 30 20:03:14.817511 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817511    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:03:14.817727 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817711    2577 factory.go:153] Registering CRI-O factory
Oct 30 20:03:14.817727 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817727    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:03:14.817727 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.816572    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-189.us-east-2.compute.internal.18e3673e82d3194c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-189.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-189.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:03:14.804095308 +0000 UTC m=+0.311517755,LastTimestamp:2026-10-30 20:03:14.804095308 +0000 UTC m=+0.311517755,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-189.us-east-2.compute.internal,}"
Oct 30 20:03:14.817917 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817789    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:03:14.817917 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817809    2577 factory.go:103] Registering Raw factory
Oct 30 20:03:14.817917 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.817823    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:03:14.818840 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.818821    2577 manager.go:319] Starting recovery of all containers
Oct 30 20:03:14.821829 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.821785    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-189.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:03:14.823533 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.823512    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phgcf"
Oct 30 20:03:14.830740 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.830719    2577 manager.go:324] Recovery completed
Oct 30 20:03:14.835549 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.835533    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:14.835628 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.835543    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 20:03:14.835834 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.835814    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phgcf"
Oct 30 20:03:14.837262 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837209    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:14.837325 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837287    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:14.837325 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837308    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:14.837905 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837893    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:03:14.837905 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837904    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:03:14.838056 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837922    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:03:14.838056 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.837898    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:14.838680 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.838666    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:14.838730 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.838687    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:14.838730 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.838698    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:14.839996 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.839919    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-189.us-east-2.compute.internal.18e3673e84cd3925  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-189.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-189.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-189.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:03:14.837264677 +0000 UTC m=+0.344687127,LastTimestamp:2026-10-30 20:03:14.837264677 +0000 UTC m=+0.344687127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-189.us-east-2.compute.internal,}"
Oct 30 20:03:14.840158 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.840146    2577 policy_none.go:50] "Start"
Oct 30 20:03:14.840214 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.840164    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:03:14.840214 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.840177    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:03:14.916822 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:14.916798    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:14.986094 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:14.986078    2577 policy_none.go:44] "Start"
Oct 30 20:03:15.013566 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.013519    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:03:15.016492 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.016478    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:03:15.017197 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.017183    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:03:15.017235 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.017198    2577 server.go:87] "Starting device plugin registration server"
Oct 30 20:03:15.017407 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.017397    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:03:15.017454 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.017408    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:03:15.017522 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.017508    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:03:15.017608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.017596    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:03:15.017608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.017608    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:03:15.018133 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.018116    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:03:15.018202 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.018158    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.046880 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.046861    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-189.us-east-2.compute.internal\" not found" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.096753 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.096732    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:03:15.098756 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.097816    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:03:15.098756 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.097836    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:03:15.098756 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.097854    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:03:15.098756 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.097885    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:03:15.102844 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.102829    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:03:15.117973 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.117960    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:15.118760 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.118743    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:15.118828 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.118771    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:15.118828 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.118782    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:15.118828 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.118803    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.128182 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.128167    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.128267 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.128187    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-189.us-east-2.compute.internal\": node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.157265 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.157246    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.198839 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.198819    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"]
Oct 30 20:03:15.198898 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.198871    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:15.199601 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.199586    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:15.199672 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.199613    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:15.199672 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.199624    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:15.201786 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.201774    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:15.201940 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.201927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.201978 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.201955    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:15.202485 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.202468    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:15.202562 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.202497    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:15.202562 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.202508    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:15.202562 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.202514    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:15.202562 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.202532    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:15.202562 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.202541    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:15.205159 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.205145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.205209 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.205169    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:03:15.205912 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.205896    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:03:15.205968 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.205927    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:03:15.205968 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.205942    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:03:15.218761 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.218746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.218824 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.218769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.218824 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.218786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8aa2772f25e2a2c0dd3af5f9c8bf9b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"d8aa2772f25e2a2c0dd3af5f9c8bf9b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.244492 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.244478    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.247113 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.247097    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.258001 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.257984    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.319656 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.319635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.319741 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.319664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.319741 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.319689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8aa2772f25e2a2c0dd3af5f9c8bf9b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"d8aa2772f25e2a2c0dd3af5f9c8bf9b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.319829 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.319750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.319829 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.319750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.319829 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.319752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8aa2772f25e2a2c0dd3af5f9c8bf9b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"d8aa2772f25e2a2c0dd3af5f9c8bf9b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.358896 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.358877    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.458938 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.458915    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.547287 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.547241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.550261 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.550247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.559312 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.559294    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.659876 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.659849    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.745290 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.745268    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:03:15.745650 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.745374    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:03:15.760554 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:15.760535    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:03:15.804122 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.804073    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:03:15.804122 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.804097    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:03:15.815643 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.815628    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:03:15.816734 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.816719    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.829202 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.829183    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:03:15.830762 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.830746    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:03:15.833247 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.833219    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:03:15.840142 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.840126    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:03:15.842266 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.842233    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:58:15 +0000 UTC" deadline="2028-05-04 00:41:23.73456828 +0000 UTC"
Oct 30 20:03:15.842303 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.842266    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13228h38m7.89230453s"
Oct 30 20:03:15.912331 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.912308    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:03:15.912442 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.912424    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hzmz7"
Oct 30 20:03:15.923996 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:15.923970    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hzmz7"
Oct 30 20:03:16.239143 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:16.238970    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aa2772f25e2a2c0dd3af5f9c8bf9b9.slice/crio-459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548 WatchSource:0}: Error finding container 459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548: Status 404 returned error can't find the container with id 459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548
Oct 30 20:03:16.245752 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.245393    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:03:16.250252 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:16.250230    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9352f491f87c9507c27b139b59fc75b2.slice/crio-93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7 WatchSource:0}: Error finding container 93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7: Status 404 returned error can't find the container with id 93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7
Oct 30 20:03:16.710141 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.710112    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:03:16.786179 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.786125    2577 apiserver.go:52] "Watching apiserver"
Oct 30 20:03:16.797559 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.797529    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:03:16.798365 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.798339    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-pnfzm","openshift-multus/network-metrics-daemon-swsxv","kube-system/konnectivity-agent-ltkv6","openshift-image-registry/node-ca-nbkgn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-nr8hm","openshift-ovn-kubernetes/ovnkube-node-lscdq","openshift-dns/node-resolver-tdk9m","openshift-multus/multus-additional-cni-plugins-t42c6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8","openshift-multus/multus-fbt2z","openshift-network-operator/iptables-alerter-fc6pd","kube-system/global-pull-secret-syncer-sxthj"]
Oct 30 20:03:16.801127 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.801100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.804168 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.804147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.806672 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.806170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:16.806672 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.806259    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:16.807152 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.807132    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:03:16.807304 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.807274    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:03:16.807369 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.807325    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cfnmr\""
Oct 30 20:03:16.808938 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.808454    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.808938 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.808533    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.808938 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.808634    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnw2v\""
Oct 30 20:03:16.808938 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.808737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:16.808938 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.808850    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:16.812244 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.810788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.813782 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.813760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.816188 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.816166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.818556 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.818535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.821255 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.821229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.823990 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.823965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.826280 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.826263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:16.827891 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.827867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.827961 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.827944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-kubelet\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.827973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-systemd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.827993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4z8\" (UniqueName: \"kubernetes.io/projected/fc397c8b-8550-4340-a2db-91263d9fcec0-kube-api-access-2c4z8\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.828049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysconfig\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.828202 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-ovn\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828202 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828311 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-script-lib\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828311 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-conf\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.828430 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:16.828430 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-slash\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828430 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-var-lib-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828430 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-node-log\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828647 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-config\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828647 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-env-overrides\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828647 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.828647 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-tmp\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.828647 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5r99\" (UniqueName: \"kubernetes.io/projected/de991caa-9b7f-4a8a-8fad-e18008ef4e47-kube-api-access-t5r99\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-netns\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc397c8b-8550-4340-a2db-91263d9fcec0-host\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-modprobe-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-run\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-netd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.828906 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcwp\" (UniqueName: \"kubernetes.io/projected/020e2934-45d4-4998-a84e-4709ce6a82eb-kube-api-access-rzcwp\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-system-cni-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.828996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc397c8b-8550-4340-a2db-91263d9fcec0-serviceca\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-sys\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-var-lib-kubelet\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-tuned\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmv5m\" (UniqueName: \"kubernetes.io/projected/4d414391-67f3-4f4b-aa36-022405704e89-kube-api-access-rmv5m\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e282920c-4244-4ffd-87a8-a21d8c96d575-tmp-dir\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtqh\" (UniqueName: \"kubernetes.io/projected/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-kube-api-access-9jtqh\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.829348 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-etc-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.829356    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/020e2934-45d4-4998-a84e-4709ce6a82eb-ovn-node-metrics-cert\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e282920c-4244-4ffd-87a8-a21d8c96d575-hosts-file\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-binary-copy\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a98a09f0-d4a5-41b5-8af3-06d949d06104-agent-certs\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a98a09f0-d4a5-41b5-8af3-06d949d06104-konnectivity-ca\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-kubernetes\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-systemd\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-systemd-units\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.829835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-log-socket\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.830496 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-bin\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.830496 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cnibin\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.830496 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-lib-modules\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.830496 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.829957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-host\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.830496 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.830041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tkm\" (UniqueName: \"kubernetes.io/projected/e282920c-4244-4ffd-87a8-a21d8c96d575-kube-api-access-67tkm\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.915968 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.915930    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.916110 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.916057    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.916185 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.916146    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:03:16.916290 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.916275    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.916425 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.916411    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.916754 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.916736    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:03:16.917141 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917125    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:03:16.917253 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917164    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:03:16.917253 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917231    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.917379 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917324    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:03:16.917438 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917429    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.917495 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917476    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.917613 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917564    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:03:16.917701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917668    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:03:16.917850 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917825    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.917926 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.917874    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.918076 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918059    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.918169 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918133    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:03:16.918227 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918171    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.918422 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918379    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.918512 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918438    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rlckg\""
Oct 30 20:03:16.918512 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918488    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-26gph\""
Oct 30 20:03:16.918629 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918549    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hms2c\""
Oct 30 20:03:16.918629 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918598    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hntbg\""
Oct 30 20:03:16.918737 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918658    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:03:16.918737 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918697    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rx62b\""
Oct 30 20:03:16.918840 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918787    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-84sqw\""
Oct 30 20:03:16.918898 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918843    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:03:16.918898 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.918885    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-phqsm\""
Oct 30 20:03:16.919159 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.919141    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:03:16.919924 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.919903    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:03:16.927448 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.927355    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:58:16 +0000 UTC" deadline="2028-08-06 17:48:27.300719964 +0000 UTC"
Oct 30 20:03:16.927448 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.927402    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15501h45m10.37332163s"
Oct 30 20:03:16.930650 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc397c8b-8550-4340-a2db-91263d9fcec0-serviceca\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.930739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-var-lib-kubelet\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.930739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.930739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtqh\" (UniqueName: \"kubernetes.io/projected/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-kube-api-access-9jtqh\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.930908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.930908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a98a09f0-d4a5-41b5-8af3-06d949d06104-konnectivity-ca\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.930908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-k8s-cni-cncf-io\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.930908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-netns\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.930908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvx94\" (UniqueName: \"kubernetes.io/projected/6afece01-c595-4d76-8a37-9911ac9c9413-kube-api-access-vvx94\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:16.930908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-socket-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-bin\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-system-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-registration-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.930988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-sys-fs\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-host\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-systemd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysconfig\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-ovn\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-cnibin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.931242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-daemon-config\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-conf\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-slash\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-var-lib-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-node-log\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-config\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qz9n\" (UniqueName: \"kubernetes.io/projected/c7c8981b-85d3-4d52-90b6-09d21f963de6-kube-api-access-8qz9n\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-cni-binary-copy\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6afece01-c595-4d76-8a37-9911ac9c9413-iptables-alerter-script\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-device-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-modprobe-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-run\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-system-cni-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-etc-kubernetes\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-dbus\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:16.931803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a98a09f0-d4a5-41b5-8af3-06d949d06104-agent-certs\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-sys\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-tuned\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmv5m\" (UniqueName: \"kubernetes.io/projected/4d414391-67f3-4f4b-aa36-022405704e89-kube-api-access-rmv5m\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e282920c-4244-4ffd-87a8-a21d8c96d575-tmp-dir\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-kubelet\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-etc-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/020e2934-45d4-4998-a84e-4709ce6a82eb-ovn-node-metrics-cert\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e282920c-4244-4ffd-87a8-a21d8c96d575-hosts-file\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-binary-copy\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.931993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-socket-dir-parent\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6afece01-c595-4d76-8a37-9911ac9c9413-host-slash\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-kubernetes\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.932666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-systemd\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-systemd-units\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-log-socket\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cnibin\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-bin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-lib-modules\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tkm\" (UniqueName: \"kubernetes.io/projected/e282920c-4244-4ffd-87a8-a21d8c96d575-kube-api-access-67tkm\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-conf-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-kubelet-config\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-kubelet\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4z8\" (UniqueName: \"kubernetes.io/projected/fc397c8b-8550-4340-a2db-91263d9fcec0-kube-api-access-2c4z8\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-script-lib\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-env-overrides\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.933550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-multus\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-tmp\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5r99\" (UniqueName: \"kubernetes.io/projected/de991caa-9b7f-4a8a-8fad-e18008ef4e47-kube-api-access-t5r99\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-netns\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-multus-certs\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc397c8b-8550-4340-a2db-91263d9fcec0-host\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-netd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcwp\" (UniqueName: \"kubernetes.io/projected/020e2934-45d4-4998-a84e-4709ce6a82eb-kube-api-access-rzcwp\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-hostroot\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.932988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkpz\" (UniqueName: \"kubernetes.io/projected/f649780e-1298-4f2b-a09f-b1527ee40c7c-kube-api-access-vvkpz\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.933547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc397c8b-8550-4340-a2db-91263d9fcec0-serviceca\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.933649    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-var-lib-kubelet\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.934441 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.933700    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934526    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a98a09f0-d4a5-41b5-8af3-06d949d06104-konnectivity-ca\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934665    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-bin\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934765    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-host\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934813    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-systemd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934864    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysconfig\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934912    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-ovn\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.934960    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935502 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.935152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-conf\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.935502 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.935204    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-slash\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935502 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.935250    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-var-lib-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935502 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.935295    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-node-log\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.935895 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.935870    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-config\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.936126 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.936105    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-modprobe-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.936257 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.936238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-run\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.936326 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.936293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.936386 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.936339    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-system-cni-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.936707 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.936688    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-sys\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.937375 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.937105    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:03:16.937375 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.937243    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e282920c-4244-4ffd-87a8-a21d8c96d575-tmp-dir\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.937375 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.937355    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-etc-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.938380 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.937935    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e282920c-4244-4ffd-87a8-a21d8c96d575-hosts-file\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:16.938465 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.938428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-binary-copy\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.938787 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.938629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.938787 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.938779    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:16.938923 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.938842    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:17.438822424 +0000 UTC m=+2.946244878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:16.938998 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.938937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-lib-modules\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.939734 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.939704    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.939839 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.939820    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-kubelet\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.940560 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.940536    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-script-lib\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.941062 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.941337 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941316    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-systemd\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.941420 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941405    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-kubernetes\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.941524 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941487    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.941571 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-log-socket\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.941602 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-systemd-units\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.941636 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cnibin\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.941999 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.941977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a98a09f0-d4a5-41b5-8af3-06d949d06104-agent-certs\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:16.942051 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.942022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-env-overrides\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.942110 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.942074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc397c8b-8550-4340-a2db-91263d9fcec0-host\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.942110 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.942075    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-netd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.942202 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.942169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-netns\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.942927 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.942900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/020e2934-45d4-4998-a84e-4709ce6a82eb-ovn-node-metrics-cert\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.944977 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.944953    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-tuned\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.946366 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.946324    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-tmp\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.950686 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.950659    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:16.950793 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.950689    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:16.950793 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.950704    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:16.950793 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:16.950772    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:17.450753022 +0000 UTC m=+2.958175462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:16.952317 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.952236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtqh\" (UniqueName: \"kubernetes.io/projected/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-kube-api-access-9jtqh\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:16.954765 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.954742    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmv5m\" (UniqueName: \"kubernetes.io/projected/4d414391-67f3-4f4b-aa36-022405704e89-kube-api-access-rmv5m\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:16.955577 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.955529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4z8\" (UniqueName: \"kubernetes.io/projected/fc397c8b-8550-4340-a2db-91263d9fcec0-kube-api-access-2c4z8\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:16.956571 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.956548    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5r99\" (UniqueName: \"kubernetes.io/projected/de991caa-9b7f-4a8a-8fad-e18008ef4e47-kube-api-access-t5r99\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:16.956922 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.956904    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcwp\" (UniqueName: \"kubernetes.io/projected/020e2934-45d4-4998-a84e-4709ce6a82eb-kube-api-access-rzcwp\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:16.957873 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:16.957850    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tkm\" (UniqueName: \"kubernetes.io/projected/e282920c-4244-4ffd-87a8-a21d8c96d575-kube-api-access-67tkm\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:17.033375 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-etc-kubernetes\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.033375 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-dbus\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:17.033375 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-kubelet\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-socket-dir-parent\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6afece01-c595-4d76-8a37-9911ac9c9413-host-slash\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-bin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-conf-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-kubelet-config\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.033621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-multus\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-multus-certs\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-hostroot\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkpz\" (UniqueName: \"kubernetes.io/projected/f649780e-1298-4f2b-a09f-b1527ee40c7c-kube-api-access-vvkpz\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-k8s-cni-cncf-io\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-netns\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx94\" (UniqueName: \"kubernetes.io/projected/6afece01-c595-4d76-8a37-9911ac9c9413-kube-api-access-vvx94\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-socket-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-system-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-registration-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-sys-fs\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-cnibin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.033966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-daemon-config\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qz9n\" (UniqueName: \"kubernetes.io/projected/c7c8981b-85d3-4d52-90b6-09d21f963de6-kube-api-access-8qz9n\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-cni-binary-copy\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6afece01-c595-4d76-8a37-9911ac9c9413-iptables-alerter-script\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.034107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-device-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034206    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-device-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-etc-kubernetes\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034312    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-dbus\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.034399    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.034465    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:03:17.534445705 +0000 UTC m=+3.041868155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034674    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-kubelet\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034730    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-socket-dir-parent\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6afece01-c595-4d76-8a37-9911ac9c9413-host-slash\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-bin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.034951 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-conf-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035535 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.034971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-kubelet-config\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:17.035535 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035535 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-multus\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035535 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-multus-certs\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035535 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-hostroot\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035535 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035495    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.035835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035546    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-k8s-cni-cncf-io\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035835 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-netns\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.035941 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-socket-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.036000 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.035967    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-system-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.036063 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.036022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-registration-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.036142 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.036098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-sys-fs\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.036196 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.036164    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-cnibin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.036670 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.036648    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-daemon-config\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.038307 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.037865    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-cni-binary-copy\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.038307 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.038261    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6afece01-c595-4d76-8a37-9911ac9c9413-iptables-alerter-script\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.053231 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.053208    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkpz\" (UniqueName: \"kubernetes.io/projected/f649780e-1298-4f2b-a09f-b1527ee40c7c-kube-api-access-vvkpz\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.053928 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.053887    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx94\" (UniqueName: \"kubernetes.io/projected/6afece01-c595-4d76-8a37-9911ac9c9413-kube-api-access-vvx94\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.055054 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.054993    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qz9n\" (UniqueName: \"kubernetes.io/projected/c7c8981b-85d3-4d52-90b6-09d21f963de6-kube-api-access-8qz9n\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.103193 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.103127    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"9352f491f87c9507c27b139b59fc75b2","Type":"ContainerStarted","Data":"93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7"}
Oct 30 20:03:17.105144 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.105113    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"d8aa2772f25e2a2c0dd3af5f9c8bf9b9","Type":"ContainerStarted","Data":"459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548"}
Oct 30 20:03:17.120389 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.120362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:17.128663 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.128636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:03:17.132487 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.132413    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98a09f0_d4a5_41b5_8af3_06d949d06104.slice/crio-b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7 WatchSource:0}: Error finding container b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7: Status 404 returned error can't find the container with id b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7
Oct 30 20:03:17.138873 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.138809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:03:17.139831 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.139806    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde991caa_9b7f_4a8a_8fad_e18008ef4e47.slice/crio-e56c8624da790cbf30c4d3c2ab867e3983e2e9fd775000050754f4c4164e8540 WatchSource:0}: Error finding container e56c8624da790cbf30c4d3c2ab867e3983e2e9fd775000050754f4c4164e8540: Status 404 returned error can't find the container with id e56c8624da790cbf30c4d3c2ab867e3983e2e9fd775000050754f4c4164e8540
Oct 30 20:03:17.149235 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.149210    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc397c8b_8550_4340_a2db_91263d9fcec0.slice/crio-946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144 WatchSource:0}: Error finding container 946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144: Status 404 returned error can't find the container with id 946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144
Oct 30 20:03:17.162887 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.162864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:03:17.172466 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.172446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:17.174234 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.174212    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode282920c_4244_4ffd_87a8_a21d8c96d575.slice/crio-f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b WatchSource:0}: Error finding container f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b: Status 404 returned error can't find the container with id f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b
Oct 30 20:03:17.184031 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.183997    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020e2934_45d4_4998_a84e_4709ce6a82eb.slice/crio-f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4 WatchSource:0}: Error finding container f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4: Status 404 returned error can't find the container with id f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4
Oct 30 20:03:17.194699 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.194677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbt2z"
Oct 30 20:03:17.198328 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.198307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:03:17.201565 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.201543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:03:17.203911 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.203889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:03:17.217422 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.217399    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afece01_c595_4d76_8a37_9911ac9c9413.slice/crio-60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e WatchSource:0}: Error finding container 60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e: Status 404 returned error can't find the container with id 60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e
Oct 30 20:03:17.221062 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:03:17.221037    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cbde63_a1b5_4cd7_8700_1b9238fd7aad.slice/crio-b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b WatchSource:0}: Error finding container b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b: Status 404 returned error can't find the container with id b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b
Oct 30 20:03:17.537846 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.537816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:17.538013 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.537867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:17.538013 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.537896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:17.538161 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538003    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:17.538161 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538090    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:03:18.538066204 +0000 UTC m=+4.045488654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:17.538161 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538108    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:17.538302 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538168    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:18.538150336 +0000 UTC m=+4.045572776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:17.538302 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538019    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:17.538302 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538205    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:17.538302 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538221    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:17.538302 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:17.538259    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:18.538246749 +0000 UTC m=+4.045669192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:17.928195 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.928101    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:58:16 +0000 UTC" deadline="2028-06-11 15:30:44.625994312 +0000 UTC"
Oct 30 20:03:17.928195 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:17.928163    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14155h27m26.697835606s"
Oct 30 20:03:18.098565 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.098493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:18.098731 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.098651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:18.099395 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.099236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:18.099395 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.099355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:18.120009 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.119972    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fc6pd" event={"ID":"6afece01-c595-4d76-8a37-9911ac9c9413","Type":"ContainerStarted","Data":"60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e"}
Oct 30 20:03:18.151216 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.151174    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nbkgn" event={"ID":"fc397c8b-8550-4340-a2db-91263d9fcec0","Type":"ContainerStarted","Data":"946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144"}
Oct 30 20:03:18.159286 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.159253    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8" event={"ID":"c7c8981b-85d3-4d52-90b6-09d21f963de6","Type":"ContainerStarted","Data":"4fd4a4f9ae2e7071a7c79b7bdf13207605c25a753b08a636f7ae9a56ee561fcd"}
Oct 30 20:03:18.171553 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.171524    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbt2z" event={"ID":"f649780e-1298-4f2b-a09f-b1527ee40c7c","Type":"ContainerStarted","Data":"4f47c97af9564267dd5c8ffbe1bf61dcbcb1c73a7afe73a0a92491ae07a1d5fe"}
Oct 30 20:03:18.179120 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.179076    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4"}
Oct 30 20:03:18.182583 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.182559    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tdk9m" event={"ID":"e282920c-4244-4ffd-87a8-a21d8c96d575","Type":"ContainerStarted","Data":"f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b"}
Oct 30 20:03:18.191767 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.191727    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pnfzm" event={"ID":"de991caa-9b7f-4a8a-8fad-e18008ef4e47","Type":"ContainerStarted","Data":"e56c8624da790cbf30c4d3c2ab867e3983e2e9fd775000050754f4c4164e8540"}
Oct 30 20:03:18.195616 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.195593    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ltkv6" event={"ID":"a98a09f0-d4a5-41b5-8af3-06d949d06104","Type":"ContainerStarted","Data":"b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7"}
Oct 30 20:03:18.197557 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.197533    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerStarted","Data":"b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b"}
Oct 30 20:03:18.546400 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.546357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:18.546513 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.546432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:18.546513 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:18.546488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:18.546637 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.546602    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:18.546689 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.546657    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:03:20.54663957 +0000 UTC m=+6.054062022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:18.547101 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.547081    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:18.547228 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.547107    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:18.547228 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.547120    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:18.547228 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.547163    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:20.54714847 +0000 UTC m=+6.054570920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:18.547415 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.547260    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:18.547415 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:18.547325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:20.54731451 +0000 UTC m=+6.054736945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:19.100972 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:19.100889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:19.101594 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:19.101017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:19.209736 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:19.208687    2577 generic.go:356] "Generic (PLEG): container finished" podID="9352f491f87c9507c27b139b59fc75b2" containerID="b3ad3f5066459f6545fe5d86dd2ebd94524c5cdff6a6664618bb949b56ce68ec" exitCode=0
Oct 30 20:03:19.209736 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:19.208786    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"9352f491f87c9507c27b139b59fc75b2","Type":"ContainerDied","Data":"b3ad3f5066459f6545fe5d86dd2ebd94524c5cdff6a6664618bb949b56ce68ec"}
Oct 30 20:03:19.224311 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:19.223583    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"d8aa2772f25e2a2c0dd3af5f9c8bf9b9","Type":"ContainerStarted","Data":"233e58115069487ddcc6e6b027d581319e354bbb3eebb3a35311b1a87e7eec4d"}
Oct 30 20:03:20.098884 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.098618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:20.099074 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.099014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:20.099504 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.099482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:20.099685 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.099592    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:20.242870 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.242776    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"9352f491f87c9507c27b139b59fc75b2","Type":"ContainerStarted","Data":"f6537648eef7cf1a75c814399bd1de4aedb2a4be95f98a3e8f79a7ba7d5bc1fa"}
Oct 30 20:03:20.263611 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.263504    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal" podStartSLOduration=4.263482192 podStartE2EDuration="4.263482192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:19.250277722 +0000 UTC m=+4.757700184" watchObservedRunningTime="2026-10-30 20:03:20.263482192 +0000 UTC m=+5.770904652"
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.569830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.569881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:20.569903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570006    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570019    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570041    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570091    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:24.570078485 +0000 UTC m=+10.077500935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570437    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570489    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:24.57047273 +0000 UTC m=+10.077895169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570548    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:20.570617 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:20.570574    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:03:24.570564419 +0000 UTC m=+10.077986854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:21.099194 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:21.099163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:21.099368 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:21.099307    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:21.256782 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:21.256530    2577 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="29b851dd8cd8f59bdd5541798f4320972fa98e3d35bd14bf5f9b688f63e676fb" exitCode=0
Oct 30 20:03:21.256782 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:21.256737    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerDied","Data":"29b851dd8cd8f59bdd5541798f4320972fa98e3d35bd14bf5f9b688f63e676fb"}
Oct 30 20:03:21.289864 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:21.289702    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" podStartSLOduration=5.289685811 podStartE2EDuration="5.289685811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:20.264388822 +0000 UTC m=+5.771811282" watchObservedRunningTime="2026-10-30 20:03:21.289685811 +0000 UTC m=+6.797108272"
Oct 30 20:03:22.098780 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:22.098744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:22.098980 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:22.098940    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:22.099409 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:22.099388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:22.099521 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:22.099498    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:22.271537 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:22.271495    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ltkv6" event={"ID":"a98a09f0-d4a5-41b5-8af3-06d949d06104","Type":"ContainerStarted","Data":"ff3b0851ae3ddd27a8d4e42ec0af298ce7978cf9695df462f35a0bc93ab748c8"}
Oct 30 20:03:22.292813 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:22.292755    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ltkv6" podStartSLOduration=3.087735514 podStartE2EDuration="7.292737008s" totalImagesPullingTime="4.205001494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:22.291830415 +0000 UTC m=+7.799252890" watchObservedRunningTime="2026-10-30 20:03:22.292737008 +0000 UTC m=+7.800159470"
Oct 30 20:03:22.720568 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:22.720492    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:22.721774 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:22.721749    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:23.100934 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:23.100862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:23.101122 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:23.100996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:23.279481 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:23.279449    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nbkgn" event={"ID":"fc397c8b-8550-4340-a2db-91263d9fcec0","Type":"ContainerStarted","Data":"a91e74f77742751805034be8d2b1c71f472a0a8a22b6441c1310ff3f20a7d383"}
Oct 30 20:03:23.279962 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:23.279497    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:23.280212 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:23.280168    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:03:23.319639 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:23.319581    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nbkgn" podStartSLOduration=2.126775121 podStartE2EDuration="7.319564401s" totalImagesPullingTime="5.19278928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:23.299882313 +0000 UTC m=+8.807304773" watchObservedRunningTime="2026-10-30 20:03:23.319564401 +0000 UTC m=+8.826986861"
Oct 30 20:03:24.098586 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:24.098508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:24.098739 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.098649    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:24.098739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:24.098708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:24.098866 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.098784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:24.285213 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:24.285100    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8" event={"ID":"c7c8981b-85d3-4d52-90b6-09d21f963de6","Type":"ContainerStarted","Data":"816c3db039f215cdba7eadabc89d40a47924893f19861b7f53c41407661f8815"}
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:24.606194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:24.606266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:24.606310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.606455    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.606524    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:32.606499874 +0000 UTC m=+18.113922328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.606986    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.607045    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:03:32.607027583 +0000 UTC m=+18.114450034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.607166    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.607184    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.607198    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:24.607286 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:24.607243    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:32.607228638 +0000 UTC m=+18.114651089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:25.100232 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:25.100208    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:25.100355 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:25.100331    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:25.264087 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:25.264051    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:03:25.291260 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:25.291169    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8" event={"ID":"c7c8981b-85d3-4d52-90b6-09d21f963de6","Type":"ContainerStarted","Data":"5b3e68f9930a9917e41994be03fe464b33841ffac51c907683600b840f6f68d1"}
Oct 30 20:03:26.031242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.030436    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:03:25.264069873Z","UUID":"8bfd4412-629d-46d4-8b18-7b03e9e522e1","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:03:26.034751 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.034146    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:03:26.034751 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.034173    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:03:26.098054 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.098032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:26.098054 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.098045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:26.098244 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:26.098211    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:26.098330 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:26.098300    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:26.295796 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.295263    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbt2z" event={"ID":"f649780e-1298-4f2b-a09f-b1527ee40c7c","Type":"ContainerStarted","Data":"8ef48962a3683790e4c2d671328f240f84cf8160d5a186b58e2d65ab1f802a33"}
Oct 30 20:03:26.297550 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.297511    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tdk9m" event={"ID":"e282920c-4244-4ffd-87a8-a21d8c96d575","Type":"ContainerStarted","Data":"5da9b822b878a5985982072f598c183ac9084177e42fa8c78fadf5d85fb92adb"}
Oct 30 20:03:26.300648 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.300622    2577 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="011169baea19c8113bcc2ae8cdba1035b3e1d0f170ba753593692d8e3fc2ff3f" exitCode=0
Oct 30 20:03:26.300777 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.300687    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerDied","Data":"011169baea19c8113bcc2ae8cdba1035b3e1d0f170ba753593692d8e3fc2ff3f"}
Oct 30 20:03:26.320084 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:26.320031    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fbt2z" podStartSLOduration=1.9057911889999999 podStartE2EDuration="10.320015089s" totalImagesPullingTime="8.4142239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:26.319184775 +0000 UTC m=+11.826607237" watchObservedRunningTime="2026-10-30 20:03:26.320015089 +0000 UTC m=+11.827437550"
Oct 30 20:03:27.099419 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.099217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:27.099556 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:27.099510    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:27.305594 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.305524    2577 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="01db30386a89f670e04db0de48659e53d30c0e0490addfb1724ac8158769d635" exitCode=0
Oct 30 20:03:27.305972 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.305610    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerDied","Data":"01db30386a89f670e04db0de48659e53d30c0e0490addfb1724ac8158769d635"}
Oct 30 20:03:27.308204 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.308175    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fc6pd" event={"ID":"6afece01-c595-4d76-8a37-9911ac9c9413","Type":"ContainerStarted","Data":"6ab4748aa19909baf25dba83270c29d16b2af9e09fc280715b446423c8321fdf"}
Oct 30 20:03:27.311226 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.311206    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8" event={"ID":"c7c8981b-85d3-4d52-90b6-09d21f963de6","Type":"ContainerStarted","Data":"141352b36ce36d2d4e7a41e84662a2d9e5fee617798844e8e3962401a5186140"}
Oct 30 20:03:27.331193 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.331157    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tdk9m" podStartSLOduration=2.6133037359999998 podStartE2EDuration="11.331144954s" totalImagesPullingTime="8.717841218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:26.372186748 +0000 UTC m=+11.879609209" watchObservedRunningTime="2026-10-30 20:03:27.331144954 +0000 UTC m=+12.838567410"
Oct 30 20:03:27.355666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:27.355619    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8" podStartSLOduration=2.538470472 podStartE2EDuration="11.355602281s" totalImagesPullingTime="8.817131809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.35499782 +0000 UTC m=+12.862420278" watchObservedRunningTime="2026-10-30 20:03:27.355602281 +0000 UTC m=+12.863024744"
Oct 30 20:03:28.098105 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:28.098078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:28.098225 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:28.098197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:28.098557 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:28.098539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:28.098661 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:28.098644    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:28.314381 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:28.314321    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pnfzm" event={"ID":"de991caa-9b7f-4a8a-8fad-e18008ef4e47","Type":"ContainerStarted","Data":"ec53ef7afa6b05f55d819e2e6323da3cc993f50fc7a4a1af4a00b7b324b37c2c"}
Oct 30 20:03:28.336852 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:28.336811    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fc6pd" podStartSLOduration=3.817219661 podStartE2EDuration="12.336798835s" totalImagesPullingTime="8.519579174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.373670617 +0000 UTC m=+12.881093064" watchObservedRunningTime="2026-10-30 20:03:28.336798835 +0000 UTC m=+13.844221291"
Oct 30 20:03:28.336974 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:28.336951    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pnfzm" podStartSLOduration=2.381393999 podStartE2EDuration="13.336946059s" totalImagesPullingTime="10.95555206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:28.336471931 +0000 UTC m=+13.843894388" watchObservedRunningTime="2026-10-30 20:03:28.336946059 +0000 UTC m=+13.844368515"
Oct 30 20:03:29.099076 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:29.099031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:29.099245 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:29.099165    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:29.319259 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:29.319216    2577 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="db545857a722cc3467113879a32ca4d735d03db1e82f2b3027adf47128bb8880" exitCode=0
Oct 30 20:03:29.319806 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:29.319270    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerDied","Data":"db545857a722cc3467113879a32ca4d735d03db1e82f2b3027adf47128bb8880"}
Oct 30 20:03:30.098244 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:30.098213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:30.098406 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:30.098371    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:30.098975 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:30.098956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:30.099092 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:30.099069    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:31.098809 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:31.098774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:31.099388 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:31.098910    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:31.326642 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:31.326452    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerStarted","Data":"3fa3f5d6ec4662438a3526d822a7eeb3c65bd2f65d3ac77ca09099ca76cdcdea"}
Oct 30 20:03:32.098371 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.098343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:32.098548 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.098437    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:32.098548 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.098494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:32.098652 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.098606    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:32.330470 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.330438    2577 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="3fa3f5d6ec4662438a3526d822a7eeb3c65bd2f65d3ac77ca09099ca76cdcdea" exitCode=0
Oct 30 20:03:32.330846 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.330485    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerDied","Data":"3fa3f5d6ec4662438a3526d822a7eeb3c65bd2f65d3ac77ca09099ca76cdcdea"}
Oct 30 20:03:32.670646 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.670626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:32.670737 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.670658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:32.670737 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:32.670679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:32.670811 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670743    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:32.670811 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670761    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:32.670811 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670790    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:32.670811 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670805    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:32.670976 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670816    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:32.670976 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670807    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:48.670790921 +0000 UTC m=+34.178213369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:32.670976 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670832    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:03:48.670825812 +0000 UTC m=+34.178248246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:32.670976 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:32.670844    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:48.670837754 +0000 UTC m=+34.178260188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:33.098501 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.098470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:33.098626 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:33.098592    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:33.335264 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.334986    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"f6c136e7bb5c3d32a8cf40fbae309c83ae54bb0e759fdb69a0f8c89a0a7d1ddc"}
Oct 30 20:03:33.335685 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.335275    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"9801c38d5a41ff105070f193154a31117d5ae8beedb1a15141a609c6b08ff6d7"}
Oct 30 20:03:33.335685 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.335343    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"4dab0da061cdfb5e22e0151c1e18b28f6a064bb99484bea5949d5f5a86385729"}
Oct 30 20:03:33.335685 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.335391    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"ed4b2db62378fd6c6042b3fd7c34f13db407a506524116b4dbf4ed9d9526ee36"}
Oct 30 20:03:33.335685 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.335436    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"a925e49847e8e096f03187c6794b81cf3df9b8cc38e1f898ea360bc4a576be22"}
Oct 30 20:03:33.335685 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.335480    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"6662eca589f935130afb18af0d0f7e428d1c2f1315a949304ce5b21bc2af3ccf"}
Oct 30 20:03:33.341048 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.340999    2577 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="07d518c91e5f12e2bb1ff7f6f2e1c411360b8f775b1101dd3614f04bf71af102" exitCode=0
Oct 30 20:03:33.341151 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:33.341053    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerDied","Data":"07d518c91e5f12e2bb1ff7f6f2e1c411360b8f775b1101dd3614f04bf71af102"}
Oct 30 20:03:34.099050 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:34.099027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:34.099189 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:34.099120    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:34.099189 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:34.099179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:34.099274 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:34.099257    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:34.346450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:34.346421    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t42c6" event={"ID":"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad","Type":"ContainerStarted","Data":"61fe7d542c2e6ba4bf5399005a4742159a469bd89607ad8ca011d35836b8dc55"}
Oct 30 20:03:34.375110 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:34.375037    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t42c6" podStartSLOduration=6.996441669 podStartE2EDuration="18.375026548s" totalImagesPullingTime="11.378584879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:34.373129116 +0000 UTC m=+19.880551572" watchObservedRunningTime="2026-10-30 20:03:34.375026548 +0000 UTC m=+19.882449003"
Oct 30 20:03:35.099646 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:35.099610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:35.099802 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:35.099711    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:35.351756 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:35.351692    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"12de03928a87169976060631925fe105c4c01213f0b8117209a8563cf1bbc238"}
Oct 30 20:03:36.098395 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:36.098370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:36.098536 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:36.098402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:36.098536 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:36.098487    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:36.098647 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:36.098579    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:37.098102 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.098078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:37.098684 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:37.098206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:37.359447 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.359377    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" event={"ID":"020e2934-45d4-4998-a84e-4709ce6a82eb","Type":"ContainerStarted","Data":"d18d00b4d20315a25857b3c1fae4560525143e4912bc88e9d1336b896b6de2ab"}
Oct 30 20:03:37.359577 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.359535    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:37.359577 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.359569    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:37.359678 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.359604    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:37.375086 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.375058    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:37.375197 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.375165    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:03:37.399297 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:37.399260    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq" podStartSLOduration=5.961633716 podStartE2EDuration="21.399249756s" totalImagesPullingTime="15.43761604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:37.398827929 +0000 UTC m=+22.906250381" watchObservedRunningTime="2026-10-30 20:03:37.399249756 +0000 UTC m=+22.906672211"
Oct 30 20:03:38.098928 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.098902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:38.099407 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:38.098995    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:38.099407 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.099074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:38.099407 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:38.099150    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:38.951745 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.951521    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxthj"]
Oct 30 20:03:38.951890 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.951866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:38.952073 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:38.952050    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:38.955799 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.955774    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nr8hm"]
Oct 30 20:03:38.955907 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.955892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:38.956036 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:38.955994    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:38.956450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.956430    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-swsxv"]
Oct 30 20:03:38.956545 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:38.956533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:38.956652 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:38.956634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:40.098900 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:40.098873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:40.099264 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:40.098974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:40.950694 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:40.950671    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tdk9m_e282920c-4244-4ffd-87a8-a21d8c96d575/dns-node-resolver/0.log"
Oct 30 20:03:41.098175 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:41.098148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:41.098315 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:41.098148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:41.098315 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:41.098271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:41.098386 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:41.098347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:41.719789 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:41.719768    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nbkgn_fc397c8b-8550-4340-a2db-91263d9fcec0/node-ca/0.log"
Oct 30 20:03:42.098384 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:42.098319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:42.098532 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:42.098407    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:43.098887 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:43.098861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:43.099340 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:43.098953    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:43.099340 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:43.099053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:43.099340 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:43.099146    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:44.098947 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:44.098866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:44.099369 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:44.098969    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:45.099429 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:45.099396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:45.099904 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:45.099520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:45.099904 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:45.099525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:45.099904 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:45.099581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:46.098874 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:46.098844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:46.099059 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:46.098950    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:47.098179 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:47.098145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:47.098553 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:47.098260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:47.098553 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:47.098288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:47.098553 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:47.098362    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:48.098147 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:48.098112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:48.098413 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.098223    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:48.692354 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:48.692326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:48.692464 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:48.692359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:48.692464 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:48.692384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:48.692464 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692453    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692475    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692480    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692496    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692505    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret podName:2b57959a-5bcb-422a-bb6b-1d237cae31fb nodeName:}" failed. No retries permitted until 2026-10-30 20:04:20.6924913 +0000 UTC m=+66.199913749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret") pod "global-pull-secret-syncer-sxthj" (UID: "2b57959a-5bcb-422a-bb6b-1d237cae31fb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692507    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wj8pp for pod openshift-network-diagnostics/network-check-target-nr8hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692520    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs podName:4d414391-67f3-4f4b-aa36-022405704e89 nodeName:}" failed. No retries permitted until 2026-10-30 20:04:20.692513505 +0000 UTC m=+66.199935939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs") pod "network-metrics-daemon-swsxv" (UID: "4d414391-67f3-4f4b-aa36-022405704e89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:03:48.692580 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:48.692543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp podName:d4067fb8-8775-4c03-a987-141bf11ca059 nodeName:}" failed. No retries permitted until 2026-10-30 20:04:20.692532526 +0000 UTC m=+66.199954965 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wj8pp" (UniqueName: "kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp") pod "network-check-target-nr8hm" (UID: "d4067fb8-8775-4c03-a987-141bf11ca059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:03:49.098937 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:49.098874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:49.098937 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:49.098892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:49.099333 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:49.098971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:49.099333 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:49.099088    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:50.099043 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:50.099013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:50.099496 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:50.099125    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:51.098459 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:51.098432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:51.098618 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:51.098525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:51.098665 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:51.098612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:51.098747 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:51.098727    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:52.098283 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:52.098253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:52.098653 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:52.098350    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:53.099098 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:53.099068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:53.099098 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:53.099103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:53.099492 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:53.099190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:53.099492 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:53.099310    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:54.098694 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:54.098663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:54.098873 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:54.098783    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:55.099762 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:55.099734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:55.100153 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:55.099804    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:55.100153 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:55.099841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:55.100153 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:55.099892    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:56.098973 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:56.098941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:56.099141 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:56.099038    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:57.099055 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:57.099026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:57.099516 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:57.099202    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:03:57.099516 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:57.099237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:57.099516 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:57.099303    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:58.099031 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:58.098991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:03:58.099173 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:58.099095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:03:59.098599 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:59.098571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:03:59.098749 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:59.098670    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:03:59.098793 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:03:59.098745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:03:59.098841 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:03:59.098826    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:04:00.098587 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:00.098557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:00.098939 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:04:00.098647    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:04:01.098829 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:01.098797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:01.099293 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:04:01.098913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxthj" podUID="2b57959a-5bcb-422a-bb6b-1d237cae31fb"
Oct 30 20:04:01.099293 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:01.098974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:01.099293 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:04:01.099110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-swsxv" podUID="4d414391-67f3-4f4b-aa36-022405704e89"
Oct 30 20:04:02.098693 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.098664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:02.098788 ip-10-0-141-189 kubenswrapper[2577]: E1030 20:04:02.098767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nr8hm" podUID="d4067fb8-8775-4c03-a987-141bf11ca059"
Oct 30 20:04:02.328117 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.328065    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeReady"
Oct 30 20:04:02.328411 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.328174    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:04:02.383862 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.383834    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rxgkl"]
Oct 30 20:04:02.386893 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.386876    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z6nqg"]
Oct 30 20:04:02.387015 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.386989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.389723 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.389704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.393999 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.393516    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:04:02.393999 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.393706    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:04:02.394469 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.394454    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:04:02.394469 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.394466    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:04:02.394566 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.394466    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j5tm5\""
Oct 30 20:04:02.395164 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5glt\" (UniqueName: \"kubernetes.io/projected/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-api-access-r5glt\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.395256 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-cert\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.395256 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9edbdb-60a5-479c-a0de-c63c1725725e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.395381 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xfz\" (UniqueName: \"kubernetes.io/projected/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-kube-api-access-25xfz\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.395381 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.395449 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9edbdb-60a5-479c-a0de-c63c1725725e-data-volume\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.395511 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.395463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9edbdb-60a5-479c-a0de-c63c1725725e-crio-socket\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.398026 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.397995    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:04:02.398697 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.398677    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z6nqg"]
Oct 30 20:04:02.401072 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.401053    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxgkl"]
Oct 30 20:04:02.481458 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.481433    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-68nkf"]
Oct 30 20:04:02.484211 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.484195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.493015 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.492987    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:04:02.493109 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.493034    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:04:02.496167 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.493647    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nfbvj\""
Oct 30 20:04:02.496167 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.493722    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:04:02.496167 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.494085    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:04:02.496167 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.494294    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c2z8r\""
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9edbdb-60a5-479c-a0de-c63c1725725e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xfz\" (UniqueName: \"kubernetes.io/projected/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-kube-api-access-25xfz\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-config-volume\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-tmp-dir\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9edbdb-60a5-479c-a0de-c63c1725725e-data-volume\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9edbdb-60a5-479c-a0de-c63c1725725e-crio-socket\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbdx\" (UniqueName: \"kubernetes.io/projected/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-kube-api-access-srbdx\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.496992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-metrics-tls\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.497053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5glt\" (UniqueName: \"kubernetes.io/projected/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-api-access-r5glt\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.497187 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.497090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-cert\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.498165 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.497952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.498165 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.498103    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-68nkf"]
Oct 30 20:04:02.498165 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.498122    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9edbdb-60a5-479c-a0de-c63c1725725e-crio-socket\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.498490 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.498467    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9edbdb-60a5-479c-a0de-c63c1725725e-data-volume\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.501351 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.501328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9edbdb-60a5-479c-a0de-c63c1725725e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.501533 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.501511    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-cert\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.515817 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.515794    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5glt\" (UniqueName: \"kubernetes.io/projected/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-api-access-r5glt\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.515947 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.515929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xfz\" (UniqueName: \"kubernetes.io/projected/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-kube-api-access-25xfz\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.597883 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.597824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbdx\" (UniqueName: \"kubernetes.io/projected/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-kube-api-access-srbdx\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.597883 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.597857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-metrics-tls\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.598071 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.598051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-config-volume\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.598137 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.598106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-tmp-dir\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.598438 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.598407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-tmp-dir\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.598578 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.598560    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-config-volume\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.599863 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.599845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-metrics-tls\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.610881 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.610859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbdx\" (UniqueName: \"kubernetes.io/projected/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-kube-api-access-srbdx\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.704621 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.704601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:02.706734 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.706718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:02.799442 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.799415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:02.997054 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.996908    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z6nqg"]
Oct 30 20:04:02.997920 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:02.997896    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxgkl"]
Oct 30 20:04:02.999729 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:02.999697    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cf5ad2_9b7d_40da_a3ea_e10d4bb6a5c9.slice/crio-f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575 WatchSource:0}: Error finding container f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575: Status 404 returned error can't find the container with id f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575
Oct 30 20:04:02.999930 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:02.999911    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9edbdb_60a5_479c_a0de_c63c1725725e.slice/crio-e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621 WatchSource:0}: Error finding container e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621: Status 404 returned error can't find the container with id e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621
Oct 30 20:04:03.071638 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.071619    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-68nkf"]
Oct 30 20:04:03.073078 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:03.073058    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4caf5a9_050f_4f30_a76d_6f0bff4b4409.slice/crio-26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c WatchSource:0}: Error finding container 26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c: Status 404 returned error can't find the container with id 26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c
Oct 30 20:04:03.098148 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.098121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:03.098236 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.098148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:03.104762 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.104742    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:04:03.104855 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.104787    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pxq5r\""
Oct 30 20:04:03.104855 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.104792    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:04:03.104855 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.104817    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qkmrx\""
Oct 30 20:04:03.411168 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.411135    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-68nkf" event={"ID":"c4caf5a9-050f-4f30-a76d-6f0bff4b4409","Type":"ContainerStarted","Data":"26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c"}
Oct 30 20:04:03.412114 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.412094    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z6nqg" event={"ID":"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9","Type":"ContainerStarted","Data":"f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575"}
Oct 30 20:04:03.413384 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.413363    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxgkl" event={"ID":"df9edbdb-60a5-479c-a0de-c63c1725725e","Type":"ContainerStarted","Data":"de513d1cf4447555fe60dd93a69d361260b7dc9ecb16a1780a0c00c3aac9c05c"}
Oct 30 20:04:03.413469 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.413409    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxgkl" event={"ID":"df9edbdb-60a5-479c-a0de-c63c1725725e","Type":"ContainerStarted","Data":"e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621"}
Oct 30 20:04:03.513253 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:03.513230    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tdk9m_e282920c-4244-4ffd-87a8-a21d8c96d575/dns-node-resolver/0.log"
Oct 30 20:04:04.098199 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:04.098164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:04.103486 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:04.103332    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:04:04.103486 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:04.103455    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mzsl\""
Oct 30 20:04:04.103486 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:04.103461    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:04:04.418775 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:04.418744    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxgkl" event={"ID":"df9edbdb-60a5-479c-a0de-c63c1725725e","Type":"ContainerStarted","Data":"890df3eec0e3247312f65253a54eb7e00f7b1d0e98d38a612abaa9a68cb6ab6b"}
Oct 30 20:04:04.512517 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:04.512488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nbkgn_fc397c8b-8550-4340-a2db-91263d9fcec0/node-ca/0.log"
Oct 30 20:04:05.424322 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.423842    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-68nkf" event={"ID":"c4caf5a9-050f-4f30-a76d-6f0bff4b4409","Type":"ContainerStarted","Data":"0eebae59dbf5fd89d114f75e7c23cf739ef3edd3a7dee913fac8682e09383a70"}
Oct 30 20:04:05.424322 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.423904    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-68nkf" event={"ID":"c4caf5a9-050f-4f30-a76d-6f0bff4b4409","Type":"ContainerStarted","Data":"98f88e2a97d7a13d88af6e919f8e3cf864166290a55d6f2c9ef65b6ecc88605a"}
Oct 30 20:04:05.424322 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.424057    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:05.445643 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.445298    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-68nkf" podStartSLOduration=1.4985834169999999 podStartE2EDuration="3.445285115s" totalImagesPullingTime="1.946701698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:05.444400786 +0000 UTC m=+50.951823265" watchObservedRunningTime="2026-10-30 20:04:05.445285115 +0000 UTC m=+50.952707571"
Oct 30 20:04:05.662446 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.662416    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"]
Oct 30 20:04:05.665547 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.665516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.670656 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.670622    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 20:04:05.670948 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.670925    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:04:05.670948 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.670942    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 20:04:05.671089 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.670974    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:04:05.671089 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.671025    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:04:05.671089 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.671047    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mj6x2\""
Oct 30 20:04:05.684293 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.684243    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"]
Oct 30 20:04:05.721155 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.721119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.721324 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.721171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d14bfe40-c6ab-48ff-a292-e67fc11dd857-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.721324 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.721200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjmj\" (UniqueName: \"kubernetes.io/projected/d14bfe40-c6ab-48ff-a292-e67fc11dd857-kube-api-access-qpjmj\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.721324 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.721240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.822449 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.822426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.822527 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.822493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.822527 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.822515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d14bfe40-c6ab-48ff-a292-e67fc11dd857-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.822679 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.822661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjmj\" (UniqueName: \"kubernetes.io/projected/d14bfe40-c6ab-48ff-a292-e67fc11dd857-kube-api-access-qpjmj\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.823167 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.823142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d14bfe40-c6ab-48ff-a292-e67fc11dd857-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.824667 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.824636    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.824790 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.824771    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.836060 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.835996    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjmj\" (UniqueName: \"kubernetes.io/projected/d14bfe40-c6ab-48ff-a292-e67fc11dd857-kube-api-access-qpjmj\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:05.980929 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:05.980847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:06.251251 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:06.251178    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"]
Oct 30 20:04:06.253747 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:06.253720    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14bfe40_c6ab_48ff_a292_e67fc11dd857.slice/crio-ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578 WatchSource:0}: Error finding container ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578: Status 404 returned error can't find the container with id ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578
Oct 30 20:04:06.427464 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:06.427437    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w" event={"ID":"d14bfe40-c6ab-48ff-a292-e67fc11dd857","Type":"ContainerStarted","Data":"ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578"}
Oct 30 20:04:06.428757 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:06.428735    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z6nqg" event={"ID":"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9","Type":"ContainerStarted","Data":"ec840e89eced54409f434ce38c5d5e97dfe94d86b7c6ea3747d62fb2b299efda"}
Oct 30 20:04:06.430827 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:06.430805    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxgkl" event={"ID":"df9edbdb-60a5-479c-a0de-c63c1725725e","Type":"ContainerStarted","Data":"cde758c78b8b44274a26abf6e0508c919f5b5a21fffa6f12f354efea4b630120"}
Oct 30 20:04:06.448526 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:06.448485    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z6nqg" podStartSLOduration=1.6661040649999999 podStartE2EDuration="4.448474081s" totalImagesPullingTime="2.782370016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:06.447863796 +0000 UTC m=+51.955286252" watchObservedRunningTime="2026-10-30 20:04:06.448474081 +0000 UTC m=+51.955896543"
Oct 30 20:04:06.469897 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:06.469855    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rxgkl" podStartSLOduration=2.059711945 podStartE2EDuration="4.469842642s" totalImagesPullingTime="2.410130697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:04:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:06.468621728 +0000 UTC m=+51.976044183" watchObservedRunningTime="2026-10-30 20:04:06.469842642 +0000 UTC m=+51.977265098"
Oct 30 20:04:08.438545 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:08.438514    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w" event={"ID":"d14bfe40-c6ab-48ff-a292-e67fc11dd857","Type":"ContainerStarted","Data":"1460c85bda4723d9d9af95faad4a129c032fe207bac95913f310c1074a0a9d49"}
Oct 30 20:04:08.438920 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:08.438563    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w" event={"ID":"d14bfe40-c6ab-48ff-a292-e67fc11dd857","Type":"ContainerStarted","Data":"b1d18830d3286168ef11382d25b600a627bb46fe54ef65d8670c22a41d739ede"}
Oct 30 20:04:08.460269 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:08.460224    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w" podStartSLOduration=2.05847505 podStartE2EDuration="3.460211929s" totalImagesPullingTime="1.401736879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:08.458611474 +0000 UTC m=+53.966033944" watchObservedRunningTime="2026-10-30 20:04:08.460211929 +0000 UTC m=+53.967634385"
Oct 30 20:04:09.374960 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:09.374929    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:10.081897 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.081864    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vccbk"]
Oct 30 20:04:10.085142 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.085117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.141960 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.141932    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:04:10.141960 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.141954    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bcdxj\""
Oct 30 20:04:10.142145 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.142131    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:04:10.160522 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-tls\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160633 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-textfile\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160633 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160633 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-sys\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-metrics-client-ca\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-root\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgrf\" (UniqueName: \"kubernetes.io/projected/a572a882-20a9-49fb-92cb-1f1d618ed9b0-kube-api-access-bfgrf\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160838 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.160838 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.160763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-wtmp\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.173118 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.173095    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:04:10.261511 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-tls\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-textfile\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-sys\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-metrics-client-ca\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-root\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261608 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgrf\" (UniqueName: \"kubernetes.io/projected/a572a882-20a9-49fb-92cb-1f1d618ed9b0-kube-api-access-bfgrf\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261887 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261640    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-sys\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261887 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261683    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-root\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261887 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.261887 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-wtmp\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.262102 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261933    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-textfile\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.262102 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.261982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-wtmp\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.262203 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.262187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.262306 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.262285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-metrics-client-ca\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.263823 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.263805    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.263901 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.263884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-tls\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.274739 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.274721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgrf\" (UniqueName: \"kubernetes.io/projected/a572a882-20a9-49fb-92cb-1f1d618ed9b0-kube-api-access-bfgrf\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.399382 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.399363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:10.409578 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:10.409553    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda572a882_20a9_49fb_92cb_1f1d618ed9b0.slice/crio-a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be WatchSource:0}: Error finding container a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be: Status 404 returned error can't find the container with id a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be
Oct 30 20:04:10.444752 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:10.444727    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vccbk" event={"ID":"a572a882-20a9-49fb-92cb-1f1d618ed9b0","Type":"ContainerStarted","Data":"a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be"}
Oct 30 20:04:12.145450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.145398    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"]
Oct 30 20:04:12.148682 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.148666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.153473 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.153439    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 20:04:12.153473 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.153448    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 20:04:12.154675 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.154650    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 20:04:12.154675 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.154663    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-br9crdma4q2oi\""
Oct 30 20:04:12.154852 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.154682    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 20:04:12.154852 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.154693    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 20:04:12.154852 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.154744    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rmkjc\""
Oct 30 20:04:12.162517 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.162483    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"]
Oct 30 20:04:12.174530 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174631 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174631 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpz75\" (UniqueName: \"kubernetes.io/projected/d9f5607f-deae-4832-b1ef-1ee7f095ca16-kube-api-access-lpz75\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174743 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174797 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5607f-deae-4832-b1ef-1ee7f095ca16-metrics-client-ca\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174844 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174905 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f5607f-deae-4832-b1ef-1ee7f095ca16-tmp\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174941 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.174991 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.174949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-grpc-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276116 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276116 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276314 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz75\" (UniqueName: \"kubernetes.io/projected/d9f5607f-deae-4832-b1ef-1ee7f095ca16-kube-api-access-lpz75\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276314 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276314 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5607f-deae-4832-b1ef-1ee7f095ca16-metrics-client-ca\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276314 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276314 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f5607f-deae-4832-b1ef-1ee7f095ca16-tmp\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276542 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276542 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-grpc-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.276829 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.276778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f5607f-deae-4832-b1ef-1ee7f095ca16-tmp\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.277141 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.277109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5607f-deae-4832-b1ef-1ee7f095ca16-metrics-client-ca\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.279035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.278975    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.279578 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.279175    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.279578 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.279251    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.279578 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.279416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.279578 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.279541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-grpc-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.279809 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.279720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.289729 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.289700    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz75\" (UniqueName: \"kubernetes.io/projected/d9f5607f-deae-4832-b1ef-1ee7f095ca16-kube-api-access-lpz75\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.451954 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.451879    2577 generic.go:356] "Generic (PLEG): container finished" podID="a572a882-20a9-49fb-92cb-1f1d618ed9b0" containerID="046064e720867a34eefd313b52cb188dd7c0fcdaff589d4d0ca16192ae7d4998" exitCode=0
Oct 30 20:04:12.451954 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.451935    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vccbk" event={"ID":"a572a882-20a9-49fb-92cb-1f1d618ed9b0","Type":"ContainerDied","Data":"046064e720867a34eefd313b52cb188dd7c0fcdaff589d4d0ca16192ae7d4998"}
Oct 30 20:04:12.470657 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.470636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:12.750160 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:12.750087    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"]
Oct 30 20:04:12.753672 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:12.753647    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f5607f_deae_4832_b1ef_1ee7f095ca16.slice/crio-4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb WatchSource:0}: Error finding container 4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb: Status 404 returned error can't find the container with id 4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb
Oct 30 20:04:13.458315 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:13.458266    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vccbk" event={"ID":"a572a882-20a9-49fb-92cb-1f1d618ed9b0","Type":"ContainerStarted","Data":"372bc61fe93a14794a85771bdc205bcd47741fd4e9238bffc9e1e278dad21a15"}
Oct 30 20:04:13.458803 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:13.458343    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vccbk" event={"ID":"a572a882-20a9-49fb-92cb-1f1d618ed9b0","Type":"ContainerStarted","Data":"6dd0ae3947b5189b32f1a8b4f8d0029225000984d1f66cab89f59121573023bd"}
Oct 30 20:04:13.460086 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:13.460061    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb"}
Oct 30 20:04:13.482174 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:13.482122    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vccbk" podStartSLOduration=2.325639359 podStartE2EDuration="3.482105982s" totalImagesPullingTime="1.089422111s" totalInitContainerRuntime="67.044512ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:13.481109619 +0000 UTC m=+58.988532079" watchObservedRunningTime="2026-10-30 20:04:13.482105982 +0000 UTC m=+58.989528440"
Oct 30 20:04:14.576242 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.576204    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64d548bdbc-6fsb4"]
Oct 30 20:04:14.579234 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.579210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.586178 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.586149    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:04:14.586178 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.586168    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:04:14.602297 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.602271    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64d548bdbc-6fsb4"]
Oct 30 20:04:14.612754 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.612731    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:04:14.612854 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.612791    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:04:14.614143 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.614124    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nkqt5\""
Oct 30 20:04:14.614200 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.614146    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ek4lghse18i1c\""
Oct 30 20:04:14.696804 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.696779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf9894b4-5224-45ae-b205-86ad336937bb-audit-log\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.696894 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.696826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-tls\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.696894 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.696850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.696971 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.696951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-metrics-server-audit-profiles\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.697028 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.696982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-client-certs\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.697087 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.697072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgcsk\" (UniqueName: \"kubernetes.io/projected/bf9894b4-5224-45ae-b205-86ad336937bb-kube-api-access-lgcsk\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.697130 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.697115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-client-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798237 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-metrics-server-audit-profiles\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798334 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-client-certs\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798438 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgcsk\" (UniqueName: \"kubernetes.io/projected/bf9894b4-5224-45ae-b205-86ad336937bb-kube-api-access-lgcsk\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798531 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-client-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798531 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf9894b4-5224-45ae-b205-86ad336937bb-audit-log\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798623 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-tls\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798657 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.798937 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.798921    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf9894b4-5224-45ae-b205-86ad336937bb-audit-log\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.801142 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.801122    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:04:14.801319 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.801304    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:04:14.801450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.801435    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:04:14.801506 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.801467    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ek4lghse18i1c\""
Oct 30 20:04:14.801506 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.801490    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:04:14.809334 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.809283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-metrics-server-audit-profiles\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.809918 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.809486    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.811215 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.811192    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-client-certs\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.811802 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.811784    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-client-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.812791 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.812767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-tls\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.813490 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.813473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgcsk\" (UniqueName: \"kubernetes.io/projected/bf9894b4-5224-45ae-b205-86ad336937bb-kube-api-access-lgcsk\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:14.896736 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.896712    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nkqt5\""
Oct 30 20:04:14.904993 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:14.904969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:15.179209 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.179136    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64d548bdbc-6fsb4"]
Oct 30 20:04:15.182472 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:15.182442    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9894b4_5224_45ae_b205_86ad336937bb.slice/crio-2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a WatchSource:0}: Error finding container 2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a: Status 404 returned error can't find the container with id 2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a
Oct 30 20:04:15.292209 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.292182    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"]
Oct 30 20:04:15.296583 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.296567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.305024 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.304983    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 20:04:15.306135 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.306112    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 20:04:15.306135 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.306128    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 20:04:15.306260 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.306147    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 20:04:15.307049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.306995    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 20:04:15.307185 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.307166    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bckqh\""
Oct 30 20:04:15.307960 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.307937    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 20:04:15.311696 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.311675    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"]
Oct 30 20:04:15.403141 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-serving-certs-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403245 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403287 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403320 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403354 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-federate-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403396 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403396 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jps\" (UniqueName: \"kubernetes.io/projected/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-kube-api-access-m9jps\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.403477 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.403457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-metrics-client-ca\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.433173 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.433120    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:15.477652 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.477620    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"199b7080a18a900303caed3530182e67f3a59c2524babf9c5640399032e2dc5f"}
Oct 30 20:04:15.477788 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.477700    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"f5c6e1f5e8e7ca972a522abc18df7307b59f71ea60ce698f61ce4eef8f7c9d72"}
Oct 30 20:04:15.477788 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.477750    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"e48a527a6573e3266d3d8c9db9bf4f24896fb9a8e2cfc66562ba7de8f56941dc"}
Oct 30 20:04:15.479372 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.479346    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4" event={"ID":"bf9894b4-5224-45ae-b205-86ad336937bb","Type":"ContainerStarted","Data":"2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a"}
Oct 30 20:04:15.504781 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.504752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-serving-certs-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.504944 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.504797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.504944 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.504819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505075 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.504990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505075 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-federate-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505194 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505194 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jps\" (UniqueName: \"kubernetes.io/projected/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-kube-api-access-m9jps\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505194 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-metrics-client-ca\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505966 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-serving-certs-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505966 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.505966 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.505815    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-metrics-client-ca\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.507913 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.507891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.507995 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.507891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.508067 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.508027    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.508137 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.508115    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-federate-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.518728 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.518707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jps\" (UniqueName: \"kubernetes.io/projected/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-kube-api-access-m9jps\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.611890 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.611856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:15.889751 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:15.889710    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"]
Oct 30 20:04:15.894268 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:15.894238    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb3ce4a_1765_4b82_8f0f_fd0107e77a7a.slice/crio-3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754 WatchSource:0}: Error finding container 3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754: Status 404 returned error can't find the container with id 3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754
Oct 30 20:04:16.389859 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.389823    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:04:16.396196 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.396172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.401346 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401319    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:04:16.401469 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401443    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401734    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401778    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401877    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401916    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401915    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.401966    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f2fd8\""
Oct 30 20:04:16.402035 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.402018    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-84ke3oa62g28q\""
Oct 30 20:04:16.402347 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.402093    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:04:16.402347 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.402156    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:04:16.402347 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.402195    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:04:16.402537 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.402515    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:04:16.402908 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.402888    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:04:16.406132 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.405582    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:04:16.406861 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.406840    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:04:16.414347 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.414324    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:04:16.487134 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.487097    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv" event={"ID":"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a","Type":"ContainerStarted","Data":"3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754"}
Oct 30 20:04:16.491283 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.491248    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"adeea636a6235fa68d78341f55a2ce74639864971fa6cb34517773c1bfd37c12"}
Oct 30 20:04:16.491423 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.491326    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"c802938c6cc32c4c0b0427ca8665c07f45970c7cd50a65690bacd4848163164d"}
Oct 30 20:04:16.491423 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.491368    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" event={"ID":"d9f5607f-deae-4832-b1ef-1ee7f095ca16","Type":"ContainerStarted","Data":"02a7df722bd0f243e8175cb4e1a2b32e3440e32fdd4fbc2f8d6122acf5da0be9"}
Oct 30 20:04:16.492308 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.492285    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:16.515174 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515297 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515297 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515297 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515297 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515456 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515456 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515456 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515456 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515456 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.515701 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.516114 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.516114 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.515785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.520562 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.520514    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh" podStartSLOduration=1.758505411 podStartE2EDuration="4.520496188s" totalImagesPullingTime="2.761990777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:16.519189999 +0000 UTC m=+62.026612467" watchObservedRunningTime="2026-10-30 20:04:16.520496188 +0000 UTC m=+62.027918645"
Oct 30 20:04:16.617133 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.617598 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617918    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.617975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618049 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.618039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618516 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.618106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618516 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.618177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.618516 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.618203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.619508 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.619482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.620663 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.620521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.621419 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.620855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.621508 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.621464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.622227 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.622170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.622227 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.622189    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.622374 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.622305    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.622896 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.622846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.623982 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.623578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.623982 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.623607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.623982 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.623932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.624837 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.624703    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.624837 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.624799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.624964 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.624858    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.624964 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.624884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.626580 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.626564    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.627912 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.627891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.628461 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.628442    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.631956 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.631934    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:16.734905 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:16.734568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:17.037529 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:17.036381    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:04:17.040968 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:17.040938    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff12252_1ad0_44d1_8567_ca7ec029de1a.slice/crio-610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e WatchSource:0}: Error finding container 610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e: Status 404 returned error can't find the container with id 610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e
Oct 30 20:04:17.496999 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:17.496953    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4" event={"ID":"bf9894b4-5224-45ae-b205-86ad336937bb","Type":"ContainerStarted","Data":"0a7780c26a31ed5743af222b5a36fd065cc3d17d29b11339c4e2574400a564b6"}
Oct 30 20:04:17.498544 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:17.498456    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e"}
Oct 30 20:04:17.527891 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:17.527832    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4" podStartSLOduration=1.8023673630000001 podStartE2EDuration="3.527816284s" totalImagesPullingTime="1.725448921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:17.52748771 +0000 UTC m=+63.034910167" watchObservedRunningTime="2026-10-30 20:04:17.527816284 +0000 UTC m=+63.035238741"
Oct 30 20:04:18.503557 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:18.503521    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv" event={"ID":"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a","Type":"ContainerStarted","Data":"a928765eaaa8b3844a184444a7737c11d3a6beb134ff4fe166c268f7d3414aff"}
Oct 30 20:04:18.503928 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:18.503585    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv" event={"ID":"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a","Type":"ContainerStarted","Data":"dcc63b97c804c202a97ec1fb3b1a84b4d48713f832abc6c9e04d252c5a74382c"}
Oct 30 20:04:18.503928 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:18.503616    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv" event={"ID":"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a","Type":"ContainerStarted","Data":"92be717712e2525ff92abf66cbf7cd027f1d4a123e4b13cfe3b8bde671696792"}
Oct 30 20:04:18.505084 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:18.505056    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2" exitCode=0
Oct 30 20:04:18.505179 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:18.505141    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"}
Oct 30 20:04:18.530824 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:18.530785    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv" podStartSLOduration=1.255174561 podStartE2EDuration="3.530770949s" totalImagesPullingTime="2.275596388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:18.529074781 +0000 UTC m=+64.036497238" watchObservedRunningTime="2026-10-30 20:04:18.530770949 +0000 UTC m=+64.038193404"
Oct 30 20:04:20.766666 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.766624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:20.767107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.766714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:20.767107 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.766757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:20.770236 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.770212    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:04:20.770368 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.770342    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:04:20.771856 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.771838    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:04:20.779323 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.779299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:20.779433 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.779411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:20.779784 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.779762    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:04:20.790327 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.790306    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:20.814231 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.814208    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qkmrx\""
Oct 30 20:04:20.819229 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.819210    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pxq5r\""
Oct 30 20:04:20.821238 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.821218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:20.827379 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.827360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:20.915261 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.915235    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mzsl\""
Oct 30 20:04:20.922429 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:20.922406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:21.109443 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:21.109414    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxthj"]
Oct 30 20:04:21.111551 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:21.111528    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-swsxv"]
Oct 30 20:04:21.201649 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:21.201609    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nr8hm"]
Oct 30 20:04:21.206048 ip-10-0-141-189 kubenswrapper[2577]: W1030 20:04:21.206018    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4067fb8_8775_4c03_a987_141bf11ca059.slice/crio-a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4 WatchSource:0}: Error finding container a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4: Status 404 returned error can't find the container with id a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4
Oct 30 20:04:21.518938 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:21.518806    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-swsxv" event={"ID":"4d414391-67f3-4f4b-aa36-022405704e89","Type":"ContainerStarted","Data":"910c137d4af317a7bc72ade2afa6b161f4d4ffa8340b8c1acfad18420f07314c"}
Oct 30 20:04:21.522702 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:21.522674    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxthj" event={"ID":"2b57959a-5bcb-422a-bb6b-1d237cae31fb","Type":"ContainerStarted","Data":"59f8a936282eff83df4e0983f7f17645795ac3f0a3d973fdcede2467ef585405"}
Oct 30 20:04:21.524364 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:21.524335    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nr8hm" event={"ID":"d4067fb8-8775-4c03-a987-141bf11ca059","Type":"ContainerStarted","Data":"a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4"}
Oct 30 20:04:22.538085 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:22.538053    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"}
Oct 30 20:04:23.514863 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.514824    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:23.553450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.550850    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"}
Oct 30 20:04:23.553450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.550982    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"}
Oct 30 20:04:23.553450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.551058    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"}
Oct 30 20:04:23.553450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.551149    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"}
Oct 30 20:04:23.553450 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.551253    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerStarted","Data":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"}
Oct 30 20:04:23.557087 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.557059    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-swsxv" event={"ID":"4d414391-67f3-4f4b-aa36-022405704e89","Type":"ContainerStarted","Data":"23de817253ecfa4fea7149ed60704e5bf80e72977d53bf6e0be78c5079737e97"}
Oct 30 20:04:23.557248 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.557232    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-swsxv" event={"ID":"4d414391-67f3-4f4b-aa36-022405704e89","Type":"ContainerStarted","Data":"9c99a27813324ed390fcac79d85e21b2aaf1685c9a379ef963d89f7e6597ceaa"}
Oct 30 20:04:23.599922 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.599467    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.659539786 podStartE2EDuration="7.599447302s" totalImagesPullingTime="4.92506902s" totalInitContainerRuntime="14.838496ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:23.597056134 +0000 UTC m=+69.104478611" watchObservedRunningTime="2026-10-30 20:04:23.599447302 +0000 UTC m=+69.106869758"
Oct 30 20:04:23.622811 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:23.622534    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-swsxv" podStartSLOduration=66.181955322 podStartE2EDuration="1m7.62252151s" totalImagesPullingTime="1.440566188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:23.620806886 +0000 UTC m=+69.128229342" watchObservedRunningTime="2026-10-30 20:04:23.62252151 +0000 UTC m=+69.129943944"
Oct 30 20:04:26.570733 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:26.570687    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nr8hm" event={"ID":"d4067fb8-8775-4c03-a987-141bf11ca059","Type":"ContainerStarted","Data":"7cb7a6b725be06d594c2ecda0adeb2494bee27a5a03a6a8db0fd646962bb1b69"}
Oct 30 20:04:26.571185 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:26.570810    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:26.590101 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:26.590044    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nr8hm" podStartSLOduration=66.209250543 podStartE2EDuration="1m10.590026993s" totalImagesPullingTime="4.38077645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:26.589071019 +0000 UTC m=+72.096493476" watchObservedRunningTime="2026-10-30 20:04:26.590026993 +0000 UTC m=+72.097449450"
Oct 30 20:04:26.735361 ip-10-0-141-189 kubenswrapper[2577]: I1030 20:04:26.735322    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:27.047214 ip-10-0-141-189 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:04:27.058412 ip-10-0-141-189 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:04:27.058578 ip-10-0-141-189 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:04:27.058629 ip-10-0-141-189 systemd[1]: kubelet.service: Consumed 3.944s CPU time.
Oct 30 20:04:27.066689 ip-10-0-141-189 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:04:27.125352 ip-10-0-141-189 kubenswrapper[6058]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:04:27.125352 ip-10-0-141-189 kubenswrapper[6058]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:04:27.125352 ip-10-0-141-189 kubenswrapper[6058]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:04:27.125352 ip-10-0-141-189 kubenswrapper[6058]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:04:27.127342 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127273    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.127342 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127342    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127346    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127350    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127353    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127356    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127359    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127362    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127365    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127368    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127371    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127373    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127376    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127383    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127386    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127388    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127391    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127395    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127400    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127403    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.127456 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127407    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127410    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127412    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127416    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127418    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127421    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127424    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127427    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127430    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127433    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127436    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127438    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127441    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127444    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127447    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127450    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127452    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127455    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127458    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127461    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.127906 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127464    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127467    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127470    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127472    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127475    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127478    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127482    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127485    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127487    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127490    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127494    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127497    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127499    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127503    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127505    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127508    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127511    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127514    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127517    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.128392 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127519    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127522    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127525    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127528    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127532    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127536    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127539    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127542    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127545    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127548    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127551    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127553    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127556    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127559    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127561    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127564    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127567    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127569    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127572    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.128838 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127575    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127578    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127581    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127583    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127586    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127589    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127591    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127595    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127598    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127600    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127603    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127606    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127609    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127611    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127614    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127617    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127620    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127623    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127625    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127628    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.129311 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127631    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127634    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127637    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127640    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127643    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127646    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127648    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.127651    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128382    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128389    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128392    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128396    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128399    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128402    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128406    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128409    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128412    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128414    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128418    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128421    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.129847 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128424    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128427    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128429    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128432    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128435    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128437    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128441    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128444    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128446    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128449    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128452    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128454    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128459    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128464    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128467    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128470    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128473    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128476    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128479    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.130347 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128481    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128484    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128487    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128490    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128493    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128495    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128499    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128502    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128506    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128508    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128511    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128514    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128517    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128520    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128522    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128525    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128528    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128530    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128533    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.130803 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128537    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128541    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128545    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128548    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128551    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128554    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128557    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128560    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128563    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128568    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128571    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128573    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128576    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128579    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128581    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128584    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128587    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128591    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128594    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128597    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.131280 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128600    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128603    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128606    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128608    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128611    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128614    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128617    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128620    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128622    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128625    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128628    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128631    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128634    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128636    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128639    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128641    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128644    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128646    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128649    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128652    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.131804 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128654    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128657    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128660    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128662    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128665    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128668    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128670    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128673    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128677    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128680    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128683    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128686    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128689    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128691    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128695    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.128698    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129194    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129200    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129203    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129206    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.132304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129209    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129212    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129215    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129217    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129220    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129223    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129226    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129229    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129233    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129237    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129241    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129244    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129247    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129250    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129254    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129257    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129259    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129264    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129267    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.132786 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129270    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129273    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129276    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129279    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129282    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129285    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129287    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129290    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129293    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129296    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129299    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129302    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129305    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129307    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129310    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129313    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129315    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129318    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129321    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129324    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.133304 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129327    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129329    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129332    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129335    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129337    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129340    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129343    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129346    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129348    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129351    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129354    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129357    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129360    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129363    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129365    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129368    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129371    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129373    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129376    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129379    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.133968 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129381    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129391    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129394    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129397    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129402    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129405    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129408    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129411    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129414    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129416    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129419    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129422    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129425    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129428    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129430    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129433    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129436    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129439    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129442    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.134661 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129444    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129447    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129450    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129453    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129456    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129458    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129461    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129464    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129467    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129470    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129472    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129475    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129478    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129481    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129484    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129487    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129490    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129492    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129495    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.135176 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129498    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129500    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129503    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129506    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.129510    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129599    6058 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129610    6058 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129616    6058 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129621    6058 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129625    6058 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129629    6058 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129633    6058 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129637    6058 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129640    6058 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129644    6058 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129648    6058 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129651    6058 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129655    6058 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129658    6058 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129661    6058 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129664    6058 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129667    6058 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129670    6058 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:04:27.135633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129675    6058 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129680    6058 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129683    6058 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129687    6058 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129690    6058 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129694    6058 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129698    6058 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129702    6058 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129705    6058 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129708    6058 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129711    6058 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129714    6058 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129718    6058 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129721    6058 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129725    6058 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129728    6058 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129731    6058 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129734    6058 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129737    6058 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129740    6058 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129751    6058 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129755    6058 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129760    6058 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129763    6058 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:04:27.136198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129766    6058 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129771    6058 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129774    6058 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129777    6058 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129780    6058 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129783    6058 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129786    6058 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129789    6058 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129792    6058 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129797    6058 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129800    6058 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129803    6058 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129807    6058 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129810    6058 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129813    6058 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129817    6058 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129820    6058 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129823    6058 flags.go:64] FLAG: --help="false"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129827    6058 flags.go:64] FLAG: --hostname-override="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129830    6058 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129833    6058 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129836    6058 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129840    6058 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:04:27.136774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129843    6058 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129846    6058 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129851    6058 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129854    6058 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129857    6058 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129861    6058 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129864    6058 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129867    6058 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129870    6058 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129873    6058 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129876    6058 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129879    6058 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129882    6058 flags.go:64] FLAG: --lock-file=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129885    6058 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129888    6058 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129891    6058 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129896    6058 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129899    6058 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129904    6058 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129907    6058 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129910    6058 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129914    6058 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129917    6058 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129920    6058 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129925    6058 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:04:27.137359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129928    6058 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129932    6058 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129935    6058 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129938    6058 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129942    6058 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129945    6058 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129948    6058 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129951    6058 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129954    6058 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129962    6058 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129965    6058 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129968    6058 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129971    6058 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129974    6058 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129977    6058 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129980    6058 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129983    6058 flags.go:64] FLAG: --port="10250"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129987    6058 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129990    6058 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07c49397d0d488fd1"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129993    6058 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129996    6058 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.129999    6058 flags.go:64] FLAG: --register-node="true"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130015    6058 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130019    6058 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130022    6058 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130026    6058 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:04:27.137955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130029    6058 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130033    6058 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130037    6058 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130040    6058 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130043    6058 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130046    6058 flags.go:64] FLAG: --runonce="false"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130049    6058 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130053    6058 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130056    6058 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130059    6058 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130062    6058 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130065    6058 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130068    6058 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130071    6058 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130074    6058 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130079    6058 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130082    6058 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130085    6058 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130089    6058 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130092    6058 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130095    6058 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130100    6058 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130103    6058 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130106    6058 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130111    6058 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:04:27.138586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130114    6058 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130117    6058 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130120    6058 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130123    6058 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130126    6058 flags.go:64] FLAG: --v="2"
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130130    6058 flags.go:64] FLAG: --version="false"
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130135    6058 flags.go:64] FLAG: --vmodule=""
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130139    6058 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130143    6058 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130279    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130283    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130286    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130289    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130292    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130295    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130298    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130301    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130304    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130307    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130310    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130312    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.139202 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130315    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130319    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130322    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130325    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130327    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130330    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130333    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130336    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130339    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130342    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130345    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130348    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130352    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130356    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130359    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130362    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130365    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130369    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130372    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130375    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.139742 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130378    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130381    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130383    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130386    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130389    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130391    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130395    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130398    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130400    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130403    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130406    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130409    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130412    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130416    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130420    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130424    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130427    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130430    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130433    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.140226 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130436    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130439    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130442    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130445    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130447    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130450    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130453    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130455    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130458    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130461    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130464    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130467    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130470    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130473    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130476    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130479    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130481    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130484    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130487    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.140690 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130490    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130493    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130496    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130498    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130501    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130504    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130507    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130511    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130514    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130517    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130520    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130522    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130525    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130528    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130530    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130533    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130536    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130539    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130541    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130544    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.141171 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130547    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130550    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130553    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130556    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130559    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130562    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130565    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130568    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130570    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130574    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130577    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130579    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130582    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130585    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130587    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130590    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.141787 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.130617    6058 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130651    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130656    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130661    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130664    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130667    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130670    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130672    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130675    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130678    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130681    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130683    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130686    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130689    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130691    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130694    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130697    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130700    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130703    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.142492 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130706    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130709    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130712    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130715    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130718    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130720    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130723    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130726    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130729    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130731    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130735    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130738    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130740    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130744    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130747    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130751    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130756    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130759    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130762    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130765    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.143017 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130768    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130770    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130774    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130776    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130779    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130782    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130784    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130787    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130790    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130793    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130795    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130798    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130801    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130805    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130808    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130811    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130815    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130820    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130823    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.143493 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130826    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130830    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130832    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130835    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130838    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130841    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130844    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130847    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130850    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130857    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130860    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130862    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130865    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130867    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130871    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130873    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130876    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130879    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130882    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.143953 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130885    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130887    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130890    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130893    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130895    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130898    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130901    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130904    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130907    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130910    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130912    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130915    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130918    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130920    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130923    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130926    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130928    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130931    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130934    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.144499 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130937    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130940    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130943    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130947    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130950    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130952    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130955    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130958    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130960    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130963    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.130966    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.135843    6058 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.135860    6058 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.135874    6058 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.135879    6058 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135934    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135939    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135944    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135947    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135950    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.145070 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135953    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135956    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135959    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135962    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135965    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135968    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135971    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135974    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135977    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135980    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135983    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135986    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135989    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135991    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135995    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.135998    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136023    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136027    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136030    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136034    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.145587 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136036    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136039    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136042    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136045    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136048    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136051    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136053    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136057    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136060    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136063    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136066    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136068    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136071    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136074    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136077    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136079    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136082    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136085    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136088    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136091    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.146121 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136093    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136097    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136100    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136102    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136106    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136108    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136111    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136114    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136117    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136120    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136123    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136125    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136128    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136131    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136136    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136140    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136144    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136147    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136150    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136153    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.146604 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136156    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136158    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136161    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136164    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136167    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136170    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136172    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136175    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136178    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136181    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136183    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136186    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136189    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136191    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136195    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136199    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136202    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136205    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136207    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136210    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.147086 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136214    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136216    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136219    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136223    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136225    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136228    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136231    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136233    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136236    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136239    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136242    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136245    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136247    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136250    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136253    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136255    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136258    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136261    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136265    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.147599 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136268    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136271    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.136299    6058 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136457    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136462    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136465    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136468    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136471    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136475    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136477    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136480    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136483    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136486    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136489    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136492    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136494    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136497    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.148125 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136500    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136503    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136506    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136509    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136511    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136514    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136517    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136520    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136523    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136525    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136528    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136531    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136534    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136536    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136539    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136542    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136545    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136548    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136551    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.148536 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136554    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136556    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136560    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136565    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136568    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136571    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136574    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136577    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136580    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136582    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136585    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136588    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136591    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136594    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136596    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136599    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136602    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136605    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136608    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136610    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.148972 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136613    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136617    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136621    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136625    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136628    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136631    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136634    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136637    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136640    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136643    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136645    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136648    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136651    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136654    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136656    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136660    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136663    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136666    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136669    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.149474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136672    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136675    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136678    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136681    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136684    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136687    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136690    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136692    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136695    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136698    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136701    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136703    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136706    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136709    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136712    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136714    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136719    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136723    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.149933 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136726    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136730    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136733    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136736    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136739    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136742    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136744    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136747    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136750    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136753    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136757    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136760    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136763    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136766    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136772    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136775    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.150427 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.136798    6058 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136830    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136835    6058 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136838    6058 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136841    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136845    6058 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136848    6058 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136851    6058 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136854    6058 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136857    6058 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136860    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136863    6058 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136866    6058 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136869    6058 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136872    6058 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136874    6058 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136877    6058 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136880    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136883    6058 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136886    6058 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:04:27.151081 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136888    6058 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136892    6058 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136896    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136899    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136902    6058 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136904    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136907    6058 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136911    6058 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136914    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136917    6058 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136919    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136922    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136925    6058 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136928    6058 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136930    6058 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136933    6058 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136936    6058 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136938    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136941    6058 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:04:27.151849 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136944    6058 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136946    6058 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136951    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136954    6058 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136957    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136960    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136963    6058 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136965    6058 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136968    6058 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136971    6058 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136974    6058 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136976    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136979    6058 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136982    6058 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136984    6058 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136987    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136991    6058 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136995    6058 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:04:27.152608 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.136998    6058 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137000    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137019    6058 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137022    6058 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137025    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137027    6058 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137030    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137034    6058 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137037    6058 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137039    6058 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137042    6058 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137045    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137048    6058 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137051    6058 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137053    6058 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137056    6058 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137061    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137064    6058 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137067    6058 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137070    6058 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:04:27.153574 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137073    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137075    6058 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137078    6058 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137081    6058 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137084    6058 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137087    6058 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137090    6058 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137092    6058 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137095    6058 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137098    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137101    6058 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137103    6058 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137106    6058 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137109    6058 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137112    6058 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137115    6058 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137117    6058 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137120    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137122    6058 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137125    6058 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:04:27.154555 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137129    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137131    6058 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137134    6058 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137137    6058 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137140    6058 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137144    6058 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137147    6058 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137150    6058 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137154    6058 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:04:27.137157    6058 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.137216    6058 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138184    6058 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138238    6058 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138423    6058 server.go:1014] "Starting client certificate rotation"
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138563    6058 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138608    6058 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:58:15 +0000 UTC" deadline="2028-05-12 00:00:51.247079887 +0000 UTC"
Oct 30 20:04:27.155474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138644    6058 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13419h56m24.108437572s"
Oct 30 20:04:27.156344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.138903    6058 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:04:27.156344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.141145    6058 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:04:27.156344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.142471    6058 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:04:27.156344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.144232    6058 log.go:25] "Validated CRI v1 image API"
Oct 30 20:04:27.156344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.144948    6058 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:04:27.156344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.146886    6058 fs.go:135] Filesystem UUIDs: map[3da0bb67-0e66-4cba-9541-19664a699fff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ece864de-a339-4c3e-8361-383eaa27df37:/dev/nvme0n1p3]
Oct 30 20:04:27.156656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.146905    6058 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f47c97af9564267dd5c8ffbe1bf61dcbcb1c73a7afe73a0a92491ae07a1d5fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f47c97af9564267dd5c8ffbe1bf61dcbcb1c73a7afe73a0a92491ae07a1d5fe/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fd4a4f9ae2e7071a7c79b7bdf13207605c25a753b08a636f7ae9a56ee561fcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fd4a4f9ae2e7071a7c79b7bdf13207605c25a753b08a636f7ae9a56ee561fcd/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59f8a936282eff83df4e0983f7f17645795ac3f0a3d973fdcede2467ef585405/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59f8a936282eff83df4e0983f7f17645795ac3f0a3d973fdcede2467ef585405/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/910c137d4af317a7bc72ade2afa6b161f4d4ffa8340b8c1acfad18420f07314c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/910c137d4af317a7bc72ade2afa6b161f4d4ffa8340b8c1acfad18420f07314c/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4/userdata/shm major:0 minor:389 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volumes/kubernetes.io~projected/kube-api-access-rzcwp:{mountpoint:/var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volumes/kubernetes.io~projected/kube-api-access-rzcwp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b57959a-5bcb-422a-bb6b-1d237cae31fb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2b57959a-5bcb-422a-bb6b-1d237cae31fb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/config-reloader/2 major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/prometheus/15 major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~projected/kube-api-access-zcvp2:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~projected/kube-api-access-zcvp2 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~projected/tls-assets major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/config major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/web-config major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d414391-67f3-4f4b-aa36-022405704e89/volumes/kubernetes.io~projected/kube-api-access-rmv5m:{mountpoint:/var/lib/kubelet/pods/4d414391-67f3-4f4b-aa36-022405704e89/volumes/kubernetes.io~projected/kube-api-access-rmv5m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d414391-67f3-4f4b-aa36-022405704e89/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4d414391-67f3-4f4b-aa36-022405704e89/volumes/kubernetes.io~secret/metrics-certs major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6afece01-c595-4d76-8a37-9911ac9c9413/volumes/kubernetes.io~projected/kube-api-access-vvx94:{mountpoint:/var/lib/kubelet/pods/6afece01-c595-4d76-8a37-9911ac9c9413/volumes/kubernetes.io~projected/kube-api-access-vvx94 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad/volumes/kubernetes.io~projected/kube-api-access-9jtqh:{mountpoint:/var/lib/kubelet/pods/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad/volumes/kubernetes.io~projected/kube-api-access-9jtqh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~projected/kube-api-access-bfgrf:{mountpoint:/var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~projected/kube-api-access-bfgrf major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a98a09f0-d4a5-41b5-8af3-06d949d06104/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a98a09f0-d4a5-41b5-8af3-06d949d06104/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~projected/kube-api-access-lgcsk:{mountpoint:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~projected/kube-api-access-lgcsk major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4caf5a9-050f-4f30-a76d-6f0bff4b4409/volumes/kubernetes.io~projected/kube-api-access-srbdx:{mountpoint:/var/lib/kubelet/pods/c4caf5a9-050f-4f30-a76d-6f0bff4b4409/volumes/kubernetes.io~projected/kube-api-access-srbdx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4caf5a9-050f-4f30-a76d-6f0bff4b4409/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c4caf5a9-050f-4f30-a76d-6f0bff4b4409/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7c8981b-85d3-4d52-90b6-09d21f963de6/volumes/kubernetes.io~projected/kube-api-access-8qz9n:{mountpoint:/var/lib/kubelet/pods/c7c8981b-85d3-4d52-90b6-09d21f963de6/volumes/kubernetes.io~projected/kube-api-access-8qz9n major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~projected/kube-api-access-qpjmj:{mountpoint:/var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~projected/kube-api-access-qpjmj major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4067fb8-8775-4c03-a987-141bf11ca059/volumes/kubernetes.io~projected/kube-api-access-wj8pp:{mountpoint:/var/lib/kubelet/pods/d4067fb8-8775-4c03-a987-141bf11ca059/volumes/kubernetes.io~projected/kube-api-access-wj8pp major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~projected/kube-api-access-lpz75:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~projected/kube-api-access-lpz75 major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~projected/kube-api-access-t5r99:{mountpoint:/var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~projected/kube-api-access-t5r99 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df9edbdb-60a5-479c-a0de-c63c1725725e/volumes/kubernetes.io~projected/kube-api-access-r5glt:{mountpoint:/var/lib/kubelet/pods/df9edbdb-60a5-479c-a0de-c63c1725725e/volumes/kubernetes.io~projected/kube-api-access-r5glt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df9edbdb-60a5-479c-a0de-c63c1725725e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/df9edbdb-60a5-479c-a0de-c63c1725725e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e282920c-4244-4ffd-87a8-a21d8c96d575/volumes/kubernetes.io~projected/kube-api-access-67tkm:{mountpoint:/var/lib/kubelet/pods/e282920c-4244-4ffd-87a8-a21d8c96d575/volumes/kubernetes.io~projected/kube-api-access-67tkm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/volumes/kubernetes.io~projected/kube-api-access-25xfz:{mountpoint:/var/lib/kubelet/pods/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/volumes/kubernetes.io~projected/kube-api-access-25xfz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f649780e-1298-4f2b-a09f-b1527ee40c7c/volumes/kubernetes.io~projected/kube-api-access-vvkpz:{mountpoint:/var/lib/kubelet/pods/f649780e-1298-4f2b-a09f-b1527ee40c7c/volumes/kubernetes.io~projected/kube-api-access-vvkpz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc397c8b-8550-4340-a2db-91263d9fcec0/volumes/kubernetes.io~projected/kube-api-access-2c4z8:{mountpoint:/var/lib/kubelet/pods/fc397c8b-8550-4340-a2db-91263d9fcec0/volumes/kubernetes.io~projected/kube-api-access-2c4z8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~projected/kube-api-access-m9jps:{mountpoint:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~projected/kube-api-access-m9jps major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:321 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0b318b177dbe1468aeed2c78ac6359e52b3299cf9e6a80437e07dd7c20f4bae2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/70de0c9f538042aec1696aa3c507f96e34a9e2a1c02e1d8f33036b13c6eecbfe/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fb74b37c602ecd17ac790fb32e84abf62f6916ac304931b7daa72ca1ed38efe5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f38e060d17e7e4743d6d0a11ecf04e59a984ceaf7167d37a488880aa0e1f9c50/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ec49d615d7392054a847cee158c7ebdf01528aa0e9d0435ebe470b6e9d968253/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/950ddf13e68965e56a44ec766c10b880f5de715eea0f86adb475b939615920da/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/eee417749610851c441be29c367f74eedeb567d4ff97fff451a9abcf0dd1c268/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c5f881babcbdbd8d77bba0b27b95e46c5696585faa27bfeac7613ab611ea6fb8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/07523fe532a5339e9357f8d1c48af7c9fa44376c4b4e33cfcd7cb866209639e5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9aac7a06e56f967e02ecde8fd4acc187acfed8486e97215f45aa7004bbed43d9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f96a91c715cc24e22a5fa71466cdb68878bef5e2a27f3c778216f5a6f85bb592/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/b042591d2a46999dcf33621814fd7915f5a8f5db6862905dbbac446ab66ee640/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0e0aae7d50f714ddd64accdbf772ece02c5b878293226a8fd1afbed6eaaec1a6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/21f271f8fb41abe3737b3a65bc474f314f1e6aca2e5f568e2f350dca8d179757/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/fc3160eab493d1b954cb63ff94d4d3dd34dcdbda580b300d5e90fbc7075b1b81/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cad16cf9dd202c73a362819bbcc07728638eec06430f5ed6a304909075e36d78/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/018a4ee9f39b0e5f1782dd88c7da2240aae5f1d73fced78948a19697a5dae64a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1dbcb04dde4f57d513b9636d7f2d7e29560596827afffa9c5e2a44de347289a4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c3454423c4c31a47e714fe828d9ff734287b0c72d0d2d8a20439e826f8f51fd5/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a9c94250e73897877861bf410eb77f43d3561938286f3f6e8baf10a3da713cd7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0b85f6d894d64813771b6d8e6cbae86016dac3c15200a5ee591cc3eee4c5054d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/47936425510b41b7b13575273a6c5e7bab86891b8a1cf18802bd01390e4264a3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/770584f0eada47db2cb93ebfb1c88fdfee290fb9841f417416d3bd2d0c82a10d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/ec170d06b5ff5b4ff9d5c9dbdebcb69eea404cfb49fc2481777545c16ab606a6/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3ed550404c8b900464bd02c5d75ee9c3acba131b9a154d88e628f7b926e5c72d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/2ec1893ece49a0cd60b7267916076534154b572bf6ec6e3fa3671f1b4908d5db/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/0633d35948284d0ac2fef7b9e9cb11b5fc0bf66c65659dbb99766376a029dacb/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/2c70655d0da745e45a52d98352f674565a236be39c63fbb4c4ada24040f831ef/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/288ec024d2e914131b2b5d2c02298a3590c2252cdf11dce5568a968b77951bf6/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/8e4ffb515d3cec7d1acdd9c23acf0f43b410b8cd63b4d8c25517430f4c5a06a2/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/f4477ab5c0f18907eaf557094545a7ac8e103a8c75760591b7065977bf2e63a5/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/9207282f1580df6fd405b6f3db0aa00d9cdacf328f9da581f3fab93dcc6cc180/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/2f2e2726ec913a4373182b72a87a1a912652929c170964beaba803e9317bac12/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/c4ab90e81e101a52b229acec5260a37ff663aba86a5a00ec444cc6bd4d2c1295/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5a233ba030b48db377fd4c9b2734399ffe6e395c078d1318aa5a8dd754cfd14e/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/c2b7f76d9c20fd555f1cfc64c138d4c681dffaca583c8324d5ee73bcfe6491a3/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/5c04199717b1be5df857f04a5f036475289b618c45a9d3c8c82cdd45726626bf/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/7abbef43fdddab921b56250f25797be34f87483109d5bb87cba84e9992196ac5/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3c4c7220900cde8e2b4e47c1a277ca5eea0a361b3bb52c648c6499e95b5e4fb6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7f62c64ba65f945e87321715640ac2a0ef567f31b57828491536283f97bffc4b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/7cb7751333ee3f2fd05487c0c02823665d3c80123d4cb6122fc82ac4e591a8dc/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/e45175734a6f64b8a3b19dfc15aad3286fefbb72e6696451e279a13c6b6631a4/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/a37bf9e88c7db4cc32def11332d603d2f458abf3e5488bb1d0db3b9282d2ac91/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/5945d4f070c7f7bf4b01b912f3f03a1bb65b43f3a357babe0c3413ba837657b5/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/be6783025939f71e60b96ddc495acb34fcbb759f24d160c84d97389572dd7074/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/7131185f105028478d8168139e51554637d6ce2ed7df21496649423daed7f08f/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/ef4f461a102de008615026b896013d22f41e46e4505fd588ffa46f6a7c416864/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/da297c023b7e34fc53dbe4cb619390092bf89b61972c122b4923493f5dab40e8/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/6f9775b6843481e10eda7dd998a2c724de81d91fc56214c93204cc8615ab8f21/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/2c8ccb2ecbb6eb2d8769b102d5bd71db69fef99fcc085fb25131264f5f8b44f6/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/137247dd147ddd292932be37ad3afda33a2b0b24e648e27073b6436028cf248b/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/7432278accfb1696a21c027c294f2ab348b5273795e30e89c3536fd800d8378b/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/a45990e992cbf7bafb582abdeba508f1a5adf1cf31b2cf7cf1f8769cc65b7dc4/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/d8b7473fdba654c60c16bbe1c5c14adf9fd0dad3f3c14f13d6dcee2deaf8377f/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/d045d4a0ffd9d484baea90470f01bd65463e30f46321abd239c055a90de21fa2/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/ebd1bf32883bc186239d168cc348f54678eb9e5926084bf82261de05298b9dcd/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/96e36c492c4b893b5caadb9a30168b8f63f6609803461b38093b8ca87877c6c7/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/20a7d41491eda200ebf7b478fa932ed8f1b49e8fcd9784125cf7e2911a4c63f4/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/aee8a2bb634a44db75d4eb17e1909d619dfa4f9a0f26a9f4d4e1be2cc83bb991/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/64783e82579cff71def16796ee5c6f5850d3517438ec725a499f7da000b13a35/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/7702b2c799e0bef965056f0364d8ecd2092b8321f1d7422e1a84984d39f60929/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/05c7017423a5368d107d9368288826f7f14a4e8fc1afd033b66e87d1304ce796/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/efb02140f7e0089bf18f79b608692b54abb4a0038ffe5ebde8490ef2e1a718af/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/7b874cfca1358140102bae9f7ee7c91b5189870886cb2de9b57ad050a7e0c1a7/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a20505c37860eb75be9d3c794e4b36b8e49f9914463f6f8e7349f8c76fcdc54f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2e69591d97a360975d55056c26d1df5bd4cf1ede5050202d00b30eebb489e48d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/dae89dc9b7698f6f0e65c659d2dce6af3ee6a1e9819779172eacd8d5c2409b87/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a850d4648ab1322c79a75577ffa267d1427909b8ecaa65c3c42c8a9482ff4a1b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e90e16856a5c07c26e1c5565618bb713b2d699e761cc96009270bb898f105b97/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d7cfe4e0609606453847c362c3e34d01908a254a5c405b2c0792c909fea87282/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/92bfbe5c9204e0c57b41e81ba8436e548d7ad90c55b8b6a7d6f12d3af3f41412/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/be818e2403408753604e6ae1d40ca717f37f4254e9da5abc8bcbccb3f4b50e7f/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f14f323a8734fba88c4a4b668996aed2254a97fa74d6438f3e586d13178aa981/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 20:04:27.157629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.157232    6058 manager.go:217] Machine: {Timestamp:2026-10-30 20:04:27.156456559 +0000 UTC m=+0.059876581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106478 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f75405e2b77da717ea74ea87382e SystemUUID:ec22f754-05e2-b77d-a717-ea74ea87382e BootID:32b3e0c4-e7b3-40e2-852e-8071218f6f57 Filesystems:[{Device:/run/containers/storage/overlay-containers/946c45cd1d4693554005cbd65110bbb06560169f6c9b2bfa37235420cdab5144/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f47c97af9564267dd5c8ffbe1bf61dcbcb1c73a7afe73a0a92491ae07a1d5fe/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebeefccbb64929076926545fa0a025ef84218def075785b5b780ea9b9f386578/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3536ede8a9f421a5adec800c76f29b94af39a976964fe15da69ca4784a083754/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6f7292851762d1388a25acc50921a8eb7746ea79b05e090a826e71e20a0b94b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d414391-67f3-4f4b-aa36-022405704e89/volumes/kubernetes.io~projected/kube-api-access-rmv5m DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/volumes/kubernetes.io~projected/kube-api-access-25xfz DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9c68baaa5468153e30645e2b7a96eec0d8330a82a330a30cc99fc63833011f4/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df9edbdb-60a5-479c-a0de-c63c1725725e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a305cf1eef45b72e91bae8ea220f94e84ce6a2f3f93dc7479a76c47e831510be/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~projected/kube-api-access-zcvp2 DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~projected/kube-api-access-lgcsk DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fc388ce7c8b0d240db828f9620402e6d86542af10321c439e89050ac0f1048a/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b57959a-5bcb-422a-bb6b-1d237cae31fb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:379 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1f01bc3b432185d457754311367daf0c4e5cddb89de51fb48fb174171a9f89b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~projected/kube-api-access-t5r99 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59f8a936282eff83df4e0983f7f17645795ac3f0a3d973fdcede2467ef585405/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d414391-67f3-4f4b-aa36-022405704e89/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad/volumes/kubernetes.io~projected/kube-api-access-9jtqh DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8eaa86c80d02f38a814e952a6659904626c5209b7584c5eb33e3eaf734730f7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4caf5a9-050f-4f30-a76d-6f0bff4b4409/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d4067fb8-8775-4c03-a987-141bf11ca059/volumes/kubernetes.io~projected/kube-api-access-wj8pp DeviceMajor:0 DeviceMinor:384 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f649780e-1298-4f2b-a09f-b1527ee40c7c/volumes/kubernetes.io~projected/kube-api-access-vvkpz DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc397c8b-8550-4340-a2db-91263d9fcec0/volumes/kubernetes.io~projected/kube-api-access-2c4z8 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a98a09f0-d4a5-41b5-8af3-06d949d06104/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26a8be3cf35dd4d1c7f8b2a0930f2a2a79ba8c51ee88a1f862411ef261d84a4c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:280 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~projected/kube-api-access-lpz75 DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4acaf8e0a1edab5901a2057597d617e481a3bc251612535f723d4c641a1838bb/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4c4db8af1758022285e9b4c428428b1dfea9422e6cd95b89f288ba058abfdc4/userdata/shm DeviceMajor:0 DeviceMinor:389 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e282920c-4244-4ffd-87a8-a21d8c96d575/volumes/kubernetes.io~projected/kube-api-access-67tkm DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df9edbdb-60a5-479c-a0de-c63c1725725e/volumes/kubernetes.io~projected/kube-api-access-r5glt DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f52fd962e8c1f48350f9ad1e700366fbf969535c00a27f3aff8d23f7e6c93575/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60a950fe80d3ed65af8a1ac84708de3513b3d4e1accead11d1d84c7cdec47e9e/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8d1d2f2fa24ad70b77e3b0797e49d62f8d12a403e017e2d5d9c7837f7be0621/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a572a882-20a9-49fb-92cb-1f1d618ed9b0/volumes/kubernetes.io~projected/kube-api-access-bfgrf DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4caf5a9-050f-4f30-a76d-6f0bff4b4409/volumes/kubernetes.io~projected/kube-api-access-srbdx DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d14bfe40-c6ab-48ff-a292-e67fc11dd857/volumes/kubernetes.io~projected/kube-api-access-qpjmj DeviceMajor:0 DeviceMinor:253 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6afece01-c595-4d76-8a37-9911ac9c9413/volumes/kubernetes.io~projected/kube-api-access-vvx94 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fd4a4f9ae2e7071a7c79b7bdf13207605c25a753b08a636f7ae9a56ee561fcd/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/910c137d4af317a7bc72ade2afa6b161f4d4ffa8340b8c1acfad18420f07314c/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/volumes/kubernetes.io~projected/kube-api-access-m9jps DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7c8981b-85d3-4d52-90b6-09d21f963de6/volumes/kubernetes.io~projected/kube-api-access-8qz9n DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/de991caa-9b7f-4a8a-8fad-e18008ef4e47/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/020e2934-45d4-4998-a84e-4709ce6a82eb/volumes/kubernetes.io~projected/kube-api-access-rzcwp DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f5607f-deae-4832-b1ef-1ee7f095ca16/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9894b4-5224-45ae-b205-86ad336937bb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26a8be3cf35dd4d MacAddress:4e:57:de:d9:9c:b5 Speed:10000 Mtu:8901} {Name:2fc388ce7c8b0d2 MacAddress:d6:9b:7a:ba:ad:9f Speed:10000 Mtu:8901} {Name:3536ede8a9f421a MacAddress:5e:c9:93:c8:9d:cd Speed:10000 Mtu:8901} {Name:4acaf8e0a1edab5 MacAddress:fe:3c:93:c2:8e:d0 Speed:10000 Mtu:8901} {Name:59f8a936282eff8 MacAddress:ca:c7:e4:02:66:34 Speed:10000 Mtu:8901} {Name:610b64edc321a86 MacAddress:76:0c:13:b2:5e:a4 Speed:10000 Mtu:8901} {Name:910c137d4af317a MacAddress:92:af:85:32:0f:2d Speed:10000 Mtu:8901} {Name:a4c4db8af175802 MacAddress:92:ae:88:89:82:9c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ee:ac:b7:21 Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:39:02:e8:01:ca Speed:0 Mtu:8901} {Name:e8d1d2f2fa24ad7 MacAddress:26:8c:40:b1:c0:f2 Speed:10000 Mtu:8901} {Name:ebeefccbb649290 MacAddress:62:d5:83:65:68:82 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ee:ac:b7:21 Speed:0 Mtu:9001} {Name:f52fd962e8c1f48 MacAddress:12:93:29:37:ce:d2 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ca:fe:b7:98:93:6b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:e7:42:45:f7:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:04:27.157629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.157631    6058 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:04:27.157951 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.157689    6058 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:04:27.158036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158000    6058 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:04:27.158162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158036    6058 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-189.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:04:27.158203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158164    6058 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:04:27.158203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158171    6058 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:04:27.158203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158182    6058 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:04:27.158203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158200    6058 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:04:27.158314 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158280    6058 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:04:27.158436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158427    6058 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:04:27.158531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158523    6058 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:04:27.158561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158556    6058 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:04:27.158588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158566    6058 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:04:27.158588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158576    6058 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:04:27.158588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158585    6058 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:04:27.158701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.158593    6058 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:04:27.159387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.159371    6058 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:04:27.159387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.159386    6058 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:04:27.159853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.159833    6058 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:04:27.162425 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162409    6058 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:04:27.162425 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162423    6058 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:04:27.162528 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162510    6058 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:04:27.162687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162676    6058 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:04:27.162721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162706    6058 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:04:27.162803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162794    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:04:27.162832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162808    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:04:27.162832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162814    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:04:27.162832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162820    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:04:27.162832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162826    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:04:27.162832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162832    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162838    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162844    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162851    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162857    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162873    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162883    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162911    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:04:27.163015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.162917    6058 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:04:27.166208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.166196    6058 server.go:1263] "Started kubelet"
Oct 30 20:04:27.166860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.166318    6058 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:04:27.166860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.166447    6058 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:04:27.166860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.166375    6058 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:04:27.166860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.166790    6058 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:04:27.166762 ip-10-0-141-189 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:04:27.172029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.169573    6058 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:04:27.174181 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.174161    6058 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.174236    6058 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.174252    6058 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.174295    6058 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-189.us-east-2.compute.internal\" not found"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.174360    6058 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.174949    6058 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.174998    6058 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:58:16 +0000 UTC" deadline="2028-07-04 19:01:42.822985 +0000 UTC"
Oct 30 20:04:27.176047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.175028    6058 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14710h57m15.647960545s"
Oct 30 20:04:27.176338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176207    6058 factory.go:55] Registering systemd factory
Oct 30 20:04:27.176338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176224    6058 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:04:27.176569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176550    6058 factory.go:153] Registering CRI-O factory
Oct 30 20:04:27.176569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176567    6058 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:04:27.176710 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176621    6058 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:04:27.176710 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176652    6058 factory.go:103] Registering Raw factory
Oct 30 20:04:27.176710 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.176667    6058 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177093    6058 manager.go:319] Starting recovery of all containers
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177129    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14bfe40-c6ab-48ff-a292-e67fc11dd857" volumeName="kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177161    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177177    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e282920c-4244-4ffd-87a8-a21d8c96d575" volumeName="kubernetes.io/empty-dir/e282920c-4244-4ffd-87a8-a21d8c96d575-tmp-dir" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177189    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-client-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177202    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177213    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a572a882-20a9-49fb-92cb-1f1d618ed9b0" volumeName="kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177224    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/configmap/d9f5607f-deae-4832-b1ef-1ee7f095ca16-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177236    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177250    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9edbdb-60a5-479c-a0de-c63c1725725e" volumeName="kubernetes.io/projected/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-api-access-r5glt" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177263    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177275    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d414391-67f3-4f4b-aa36-022405704e89" volumeName="kubernetes.io/projected/4d414391-67f3-4f4b-aa36-022405704e89-kube-api-access-rmv5m" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177287    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4caf5a9-050f-4f30-a76d-6f0bff4b4409" volumeName="kubernetes.io/empty-dir/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-tmp-dir" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177312    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/empty-dir/d9f5607f-deae-4832-b1ef-1ee7f095ca16-tmp" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177374    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177390    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177403    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14bfe40-c6ab-48ff-a292-e67fc11dd857" volumeName="kubernetes.io/configmap/d14bfe40-c6ab-48ff-a292-e67fc11dd857-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177414    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-grpc-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177426    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de991caa-9b7f-4a8a-8fad-e18008ef4e47" volumeName="kubernetes.io/projected/de991caa-9b7f-4a8a-8fad-e18008ef4e47-kube-api-access-t5r99" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177437    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9edbdb-60a5-479c-a0de-c63c1725725e" volumeName="kubernetes.io/secret/df9edbdb-60a5-479c-a0de-c63c1725725e-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177448    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177460    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98a09f0-d4a5-41b5-8af3-06d949d06104" volumeName="kubernetes.io/secret/a98a09f0-d4a5-41b5-8af3-06d949d06104-agent-certs" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177472    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/projected/bf9894b4-5224-45ae-b205-86ad336937bb-kube-api-access-lgcsk" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177483    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9" volumeName="kubernetes.io/secret/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-cert" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177494    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/projected/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-kube-api-access-m9jps" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177505    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-federate-client-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177515    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177526    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" volumeName="kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177544    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" volumeName="kubernetes.io/projected/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-kube-api-access-9jtqh" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177564    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4caf5a9-050f-4f30-a76d-6f0bff4b4409" volumeName="kubernetes.io/configmap/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-config-volume" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177577    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14bfe40-c6ab-48ff-a292-e67fc11dd857" volumeName="kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177588    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177600    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177611    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177626    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177638    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177649    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6afece01-c595-4d76-8a37-9911ac9c9413" volumeName="kubernetes.io/configmap/6afece01-c595-4d76-8a37-9911ac9c9413-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177671    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a572a882-20a9-49fb-92cb-1f1d618ed9b0" volumeName="kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177683    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4caf5a9-050f-4f30-a76d-6f0bff4b4409" volumeName="kubernetes.io/secret/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-metrics-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177694    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/projected/d9f5607f-deae-4832-b1ef-1ee7f095ca16-kube-api-access-lpz75" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177706    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f649780e-1298-4f2b-a09f-b1527ee40c7c" volumeName="kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177722    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177735    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e2934-45d4-4998-a84e-4709ce6a82eb" volumeName="kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-config" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177766    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177777    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177789    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc397c8b-8550-4340-a2db-91263d9fcec0" volumeName="kubernetes.io/configmap/fc397c8b-8550-4340-a2db-91263d9fcec0-serviceca" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177814    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177825    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177836    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177849    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b57959a-5bcb-422a-bb6b-1d237cae31fb" volumeName="kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177881    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177893    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a572a882-20a9-49fb-92cb-1f1d618ed9b0" volumeName="kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-tls" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177907    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f5607f-deae-4832-b1ef-1ee7f095ca16" volumeName="kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177919    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9edbdb-60a5-479c-a0de-c63c1725725e" volumeName="kubernetes.io/empty-dir/df9edbdb-60a5-479c-a0de-c63c1725725e-data-volume" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177930    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e2934-45d4-4998-a84e-4709ce6a82eb" volumeName="kubernetes.io/secret/020e2934-45d4-4998-a84e-4709ce6a82eb-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:04:27.177897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177942    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177955    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177978    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.177991    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178034    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de991caa-9b7f-4a8a-8fad-e18008ef4e47" volumeName="kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-tmp" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178047    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e2934-45d4-4998-a84e-4709ce6a82eb" volumeName="kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-env-overrides" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178063    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d414391-67f3-4f4b-aa36-022405704e89" volumeName="kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178075    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178086    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4067fb8-8775-4c03-a987-141bf11ca059" volumeName="kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178097    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f649780e-1298-4f2b-a09f-b1527ee40c7c" volumeName="kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178108    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f649780e-1298-4f2b-a09f-b1527ee40c7c" volumeName="kubernetes.io/projected/f649780e-1298-4f2b-a09f-b1527ee40c7c-kube-api-access-vvkpz" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178120    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178131    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" volumeName="kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178143    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98a09f0-d4a5-41b5-8af3-06d949d06104" volumeName="kubernetes.io/configmap/a98a09f0-d4a5-41b5-8af3-06d949d06104-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178154    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178190    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14bfe40-c6ab-48ff-a292-e67fc11dd857" volumeName="kubernetes.io/projected/d14bfe40-c6ab-48ff-a292-e67fc11dd857-kube-api-access-qpjmj" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178204    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e282920c-4244-4ffd-87a8-a21d8c96d575" volumeName="kubernetes.io/projected/e282920c-4244-4ffd-87a8-a21d8c96d575-kube-api-access-67tkm" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178215    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178229    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e2934-45d4-4998-a84e-4709ce6a82eb" volumeName="kubernetes.io/projected/020e2934-45d4-4998-a84e-4709ce6a82eb-kube-api-access-rzcwp" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178252    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178263    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6afece01-c595-4d76-8a37-9911ac9c9413" volumeName="kubernetes.io/projected/6afece01-c595-4d76-8a37-9911ac9c9413-kube-api-access-vvx94" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178287    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/empty-dir/bf9894b4-5224-45ae-b205-86ad336937bb-audit-log" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178300    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de991caa-9b7f-4a8a-8fad-e18008ef4e47" volumeName="kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-tuned" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178325    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9edbdb-60a5-479c-a0de-c63c1725725e" volumeName="kubernetes.io/configmap/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178346    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9" volumeName="kubernetes.io/projected/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-kube-api-access-25xfz" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178359    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc397c8b-8550-4340-a2db-91263d9fcec0" volumeName="kubernetes.io/projected/fc397c8b-8550-4340-a2db-91263d9fcec0-kube-api-access-2c4z8" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178372    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178407    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a572a882-20a9-49fb-92cb-1f1d618ed9b0" volumeName="kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178420    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9894b4-5224-45ae-b205-86ad336937bb" volumeName="kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-client-ca-bundle" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178432    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4caf5a9-050f-4f30-a76d-6f0bff4b4409" volumeName="kubernetes.io/projected/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-kube-api-access-srbdx" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178503    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178515    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a" volumeName="kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178527    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ff12252-1ad0-44d1-8567-ca7ec029de1a" volumeName="kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178547    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e2934-45d4-4998-a84e-4709ce6a82eb" volumeName="kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178561    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" volumeName="kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178583    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a572a882-20a9-49fb-92cb-1f1d618ed9b0" volumeName="kubernetes.io/empty-dir/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-textfile" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178619    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a572a882-20a9-49fb-92cb-1f1d618ed9b0" volumeName="kubernetes.io/projected/a572a882-20a9-49fb-92cb-1f1d618ed9b0-kube-api-access-bfgrf" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178632    6058 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c8981b-85d3-4d52-90b6-09d21f963de6" volumeName="kubernetes.io/projected/c7c8981b-85d3-4d52-90b6-09d21f963de6-kube-api-access-8qz9n" seLinuxMountContext=""
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178644    6058 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.178670    6058 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:04:27.182501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.179110    6058 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:04:27.188973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.187114    6058 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:04:27.188973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.188320    6058 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:04:27.190208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.190184    6058 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:04:27.203281 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.203254    6058 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:04:27.205101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.205079    6058 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:04:27.205101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.205102    6058 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:04:27.205252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.205125    6058 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:04:27.205252 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.205172    6058 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:04:27.210422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.210397    6058 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:04:27.219193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.218751    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2" exitCode=0
Oct 30 20:04:27.236453 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.236433    6058 generic.go:356] "Generic (PLEG): container finished" podID="a572a882-20a9-49fb-92cb-1f1d618ed9b0" containerID="046064e720867a34eefd313b52cb188dd7c0fcdaff589d4d0ca16192ae7d4998" exitCode=0
Oct 30 20:04:27.245989 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.245964    6058 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="07d518c91e5f12e2bb1ff7f6f2e1c411360b8f775b1101dd3614f04bf71af102" exitCode=0
Oct 30 20:04:27.245989 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.245988    6058 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="3fa3f5d6ec4662438a3526d822a7eeb3c65bd2f65d3ac77ca09099ca76cdcdea" exitCode=0
Oct 30 20:04:27.246156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.245995    6058 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="db545857a722cc3467113879a32ca4d735d03db1e82f2b3027adf47128bb8880" exitCode=0
Oct 30 20:04:27.246156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.246020    6058 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="01db30386a89f670e04db0de48659e53d30c0e0490addfb1724ac8158769d635" exitCode=0
Oct 30 20:04:27.246156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.246028    6058 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="011169baea19c8113bcc2ae8cdba1035b3e1d0f170ba753593692d8e3fc2ff3f" exitCode=0
Oct 30 20:04:27.246156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.246034    6058 generic.go:356] "Generic (PLEG): container finished" podID="a3cbde63-a1b5-4cd7-8700-1b9238fd7aad" containerID="29b851dd8cd8f59bdd5541798f4320972fa98e3d35bd14bf5f9b688f63e676fb" exitCode=0
Oct 30 20:04:27.259361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.259336    6058 generic.go:356] "Generic (PLEG): container finished" podID="9352f491f87c9507c27b139b59fc75b2" containerID="b3ad3f5066459f6545fe5d86dd2ebd94524c5cdff6a6664618bb949b56ce68ec" exitCode=0
Oct 30 20:04:27.287905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.287883    6058 manager.go:324] Recovery completed
Oct 30 20:04:27.305814 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.305790    6058 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 20:04:27.311013 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.310985    6058 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:04:27.311013 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311000    6058 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:04:27.311104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311030    6058 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:04:27.311153 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311140    6058 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:04:27.311187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311150    6058 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:04:27.311187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311168    6058 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:04:27.311187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311173    6058 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:04:27.311187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311178    6058 policy_none.go:50] "Start"
Oct 30 20:04:27.311187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311187    6058 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:04:27.311314 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311196    6058 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:04:27.311314 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311310    6058 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:04:27.311376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311316    6058 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:04:27.311376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.311322    6058 policy_none.go:44] "Start"
Oct 30 20:04:27.315735 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.315709    6058 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:04:27.319150 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.319134    6058 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:04:27.319713 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.319157    6058 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:04:27.319713 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.319713    6058 server.go:87] "Starting device plugin registration server"
Oct 30 20:04:27.319971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.319957    6058 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:04:27.320055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.319968    6058 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:04:27.320114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.320051    6058 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:04:27.320166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.320133    6058 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:04:27.320215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.320174    6058 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:04:27.320215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.320182    6058 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:04:27.320312 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.320228    6058 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:04:27.320159793Z","UUID":"aa9b6905-bd9d-46f2-a203-f189bb4775a8","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:04:27.321927 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.321905    6058 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:04:27.322588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.322568    6058 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:04:27.322663 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.322599    6058 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:04:27.420379 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.420328    6058 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:04:27.422239 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.422221    6058 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:04:27.422330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.422252    6058 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:04:27.422330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.422263    6058 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-189.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:04:27.422330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.422300    6058 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.436081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.436061    6058 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.436151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.436111    6058 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.506433 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.506383    6058 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"]
Oct 30 20:04:27.507036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507001    6058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.507123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507089    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"9352f491f87c9507c27b139b59fc75b2","Type":"ContainerStarted","Data":"f6537648eef7cf1a75c814399bd1de4aedb2a4be95f98a3e8f79a7ba7d5bc1fa"}
Oct 30 20:04:27.507155 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507139    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"9352f491f87c9507c27b139b59fc75b2","Type":"ContainerDied","Data":"b3ad3f5066459f6545fe5d86dd2ebd94524c5cdff6a6664618bb949b56ce68ec"}
Oct 30 20:04:27.507184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507156    6058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.507184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507161    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"9352f491f87c9507c27b139b59fc75b2","Type":"ContainerStarted","Data":"93896f380c5df60cd054c7b218f59eae88d901453121fe81276eba4db67b1db7"}
Oct 30 20:04:27.507237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507194    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"d8aa2772f25e2a2c0dd3af5f9c8bf9b9","Type":"ContainerStarted","Data":"233e58115069487ddcc6e6b027d581319e354bbb3eebb3a35311b1a87e7eec4d"}
Oct 30 20:04:27.507237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.507214    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal" event={"ID":"d8aa2772f25e2a2c0dd3af5f9c8bf9b9","Type":"ContainerStarted","Data":"459cc958aeeba0f481407bf2a92a73bc691ff0eb2744349f0d0dcd95186db548"}
Oct 30 20:04:27.523067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.523050    6058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:04:27.523113 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.523087    6058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.524264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.524251    6058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:04:27.524306 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:27.524285    6058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.589583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.589561    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.589658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.589587    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.589658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.589608    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8aa2772f25e2a2c0dd3af5f9c8bf9b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"d8aa2772f25e2a2c0dd3af5f9c8bf9b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.690153 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.690104    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8aa2772f25e2a2c0dd3af5f9c8bf9b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"d8aa2772f25e2a2c0dd3af5f9c8bf9b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.690213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.690171    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.690213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.690134    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.690213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.690200    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8aa2772f25e2a2c0dd3af5f9c8bf9b9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"d8aa2772f25e2a2c0dd3af5f9c8bf9b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.690301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.690234    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:27.690301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:27.690248    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9352f491f87c9507c27b139b59fc75b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" (UID: \"9352f491f87c9507c27b139b59fc75b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:28.158971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.158946    6058 apiserver.go:52] "Watching apiserver"
Oct 30 20:04:28.169153 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.169132    6058 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:04:28.169897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.169878    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vccbk","kube-system/konnectivity-agent-ltkv6","openshift-cluster-node-tuning-operator/tuned-pnfzm","openshift-ingress-canary/ingress-canary-z6nqg","openshift-monitoring/prometheus-operator-9cd89b74c-9v99w","kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-nr8hm","openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv","openshift-ovn-kubernetes/ovnkube-node-lscdq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-fc6pd","openshift-dns/dns-default-68nkf","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-t42c6","openshift-multus/multus-fbt2z","openshift-image-registry/node-ca-nbkgn","openshift-monitoring/metrics-server-64d548bdbc-6fsb4","openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh","openshift-dns/node-resolver-tdk9m","openshift-insights/insights-runtime-extractor-rxgkl","openshift-multus/network-metrics-daemon-swsxv","kube-system/global-pull-secret-syncer-sxthj"]
Oct 30 20:04:28.177016 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.176975    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.177133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.177072    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mzsl\""
Oct 30 20:04:28.177391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.177366    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.177572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.177552    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:04:28.177774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.177756    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:04:28.178285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178257    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:04:28.178374 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178312    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:04:28.178433 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178422    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rlckg\""
Oct 30 20:04:28.178502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178482    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.178502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178492    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:04:28.178692 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178675    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:04:28.178763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178729    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-26gph\""
Oct 30 20:04:28.178821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178793    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.178888 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178835    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:04:28.179043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179025    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.179112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179039    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:04:28.179112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179061    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.179204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179162    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:04:28.179204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179167    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cfnmr\""
Oct 30 20:04:28.179331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179313    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.179608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179574    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.179608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179592    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:04:28.179940 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179919    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hntbg\""
Oct 30 20:04:28.180041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179951    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.180137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180059    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.180137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.178675    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnw2v\""
Oct 30 20:04:28.180137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.179345    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.180286 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180194    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qkmrx\""
Oct 30 20:04:28.180726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180708    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.180808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180741    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.180808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180760    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rx62b\""
Oct 30 20:04:28.180926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180863    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hms2c\""
Oct 30 20:04:28.180926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180867    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.181067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.180939    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.181169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.181133    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:04:28.181330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.181315    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:04:28.181415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.181400    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-phqsm\""
Oct 30 20:04:28.181837 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.181824    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.184417 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184397    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:04:28.184518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184409    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:04:28.184733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184712    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pxq5r\""
Oct 30 20:04:28.184810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184739    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:04:28.184810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184783    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:04:28.184810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184783    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-84sqw\""
Oct 30 20:04:28.184933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184829    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:04:28.184933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.184860    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c2z8r\""
Oct 30 20:04:28.185394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.185370    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:04:28.185394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.185382    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:04:28.185762 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.185741    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j5tm5\""
Oct 30 20:04:28.186277 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.186260    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 20:04:28.186340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.186314    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nfbvj\""
Oct 30 20:04:28.186457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.186445    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.187021 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.186993    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.187058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.187036    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.187058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.187049    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.187119 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.187082    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:04:28.187529 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.187518    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:04:28.189216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189198    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 20:04:28.189296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189213    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mj6x2\""
Oct 30 20:04:28.189296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189224    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:04:28.189296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189229    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:04:28.189510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189492    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 20:04:28.189593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189516    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 20:04:28.189668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189619    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:04:28.189668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189621    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 20:04:28.189668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189631    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 20:04:28.189895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189880    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:04:28.189990 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189939    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:04:28.189990 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.189981    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bcdxj\""
Oct 30 20:04:28.190166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190137    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:04:28.190341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190277    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 20:04:28.190341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190292    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:04:28.190341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190329    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nkqt5\""
Oct 30 20:04:28.190477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190406    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 20:04:28.190477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190422    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 20:04:28.190477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190432    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 20:04:28.190477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190442    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190638    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190645    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190657    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190670    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190674    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190721    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190783    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rmkjc\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190793    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-br9crdma4q2oi\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190799    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bckqh\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190813    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:04:28.190783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190817    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:04:28.191230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190826    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:04:28.191230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190819    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 20:04:28.191230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190864    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:04:28.191230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.190875    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:04:28.191623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191598    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-device-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.191623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191620    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-kubelet\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.191747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191636    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-conf-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.191747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191655    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.191747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191674    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcwp\" (UniqueName: \"kubernetes.io/projected/020e2934-45d4-4998-a84e-4709ce6a82eb-kube-api-access-rzcwp\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.191747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191705    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ek4lghse18i1c\""
Oct 30 20:04:28.191747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191718    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-run\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.191747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191757    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191774    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191815    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-lib-modules\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191851    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-hostroot\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191854    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f2fd8\""
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191908    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-kubelet-config\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191927    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-84ke3oa62g28q\""
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191942    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.191974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.191971    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-systemd-units\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192071    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e282920c-4244-4ffd-87a8-a21d8c96d575-hosts-file\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192100    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6afece01-c595-4d76-8a37-9911ac9c9413-iptables-alerter-script\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192150    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx94\" (UniqueName: \"kubernetes.io/projected/6afece01-c595-4d76-8a37-9911ac9c9413-kube-api-access-vvx94\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192181    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192207    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-sys\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192232    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-bin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192260    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-var-lib-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192274    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b57959a-5bcb-422a-bb6b-1d237cae31fb-original-pull-secret\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192286    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-log-socket\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192293    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6afece01-c595-4d76-8a37-9911ac9c9413-iptables-alerter-script\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192320    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-netd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192368    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-systemd\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192380    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192385    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-k8s-cni-cncf-io\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192421    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a98a09f0-d4a5-41b5-8af3-06d949d06104-agent-certs\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192441    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192487    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-modprobe-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192590    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-kubernetes\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.192640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192635    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5r99\" (UniqueName: \"kubernetes.io/projected/de991caa-9b7f-4a8a-8fad-e18008ef4e47-kube-api-access-t5r99\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192693    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-registration-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192745    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a98a09f0-d4a5-41b5-8af3-06d949d06104-agent-certs\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192769    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-sys-fs\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192798    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-multus\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192839    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192859    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-ovn\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192881    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-node-log\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192931    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6afece01-c595-4d76-8a37-9911ac9c9413-host-slash\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192950    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-socket-dir-parent\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192970    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-kubelet\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192983    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.192998    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-conf\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193059    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-systemd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193132    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193161    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-cnibin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193186    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-daemon-config\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193227    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4z8\" (UniqueName: \"kubernetes.io/projected/fc397c8b-8550-4340-a2db-91263d9fcec0-kube-api-access-2c4z8\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193248    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysconfig\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193268    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qz9n\" (UniqueName: \"kubernetes.io/projected/c7c8981b-85d3-4d52-90b6-09d21f963de6-kube-api-access-8qz9n\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193285    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-system-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193375    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-cni-binary-copy\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193415    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-etc-kubernetes\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193521    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a98a09f0-d4a5-41b5-8af3-06d949d06104-konnectivity-ca\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193551    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-script-lib\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193584    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-daemon-config\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193594    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193658    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc397c8b-8550-4340-a2db-91263d9fcec0-host\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193704    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-netns\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193740    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f649780e-1298-4f2b-a09f-b1527ee40c7c-cni-binary-copy\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193741    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-bin\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193740    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a98a09f0-d4a5-41b5-8af3-06d949d06104-konnectivity-ca\") pod \"konnectivity-agent-ltkv6\" (UID: \"a98a09f0-d4a5-41b5-8af3-06d949d06104\") " pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193786    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/020e2934-45d4-4998-a84e-4709ce6a82eb-ovn-node-metrics-cert\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193816    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-script-lib\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193886    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e282920c-4244-4ffd-87a8-a21d8c96d575-tmp-dir\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193929    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193934    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e282920c-4244-4ffd-87a8-a21d8c96d575-tmp-dir\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193965    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-netns\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.193993    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc397c8b-8550-4340-a2db-91263d9fcec0-serviceca\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194045    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-etc-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194051    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/020e2934-45d4-4998-a84e-4709ce6a82eb-ovn-node-metrics-cert\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194068    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-env-overrides\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194103    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-socket-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194128    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-host\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194154    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-multus-certs\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194206    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-dbus\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194241    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tkm\" (UniqueName: \"kubernetes.io/projected/e282920c-4244-4ffd-87a8-a21d8c96d575-kube-api-access-67tkm\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194269    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-var-lib-kubelet\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194266    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-env-overrides\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194298    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-tuned\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194326    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-tmp\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.194330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194346    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-tuned\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.196317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194383    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkpz\" (UniqueName: \"kubernetes.io/projected/f649780e-1298-4f2b-a09f-b1527ee40c7c-kube-api-access-vvkpz\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.196317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194402    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de991caa-9b7f-4a8a-8fad-e18008ef4e47-tmp\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.196317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194442    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-slash\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.196317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194518    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc397c8b-8550-4340-a2db-91263d9fcec0-serviceca\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.196317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194531    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-config\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.196317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.194685    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/020e2934-45d4-4998-a84e-4709ce6a82eb-ovnkube-config\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.196941 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.196922    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 20:04:28.200466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.197586    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:04:28.200466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.198101    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:04:28.206524 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.206495    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx94\" (UniqueName: \"kubernetes.io/projected/6afece01-c595-4d76-8a37-9911ac9c9413-kube-api-access-vvx94\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.206814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.206799    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcwp\" (UniqueName: \"kubernetes.io/projected/020e2934-45d4-4998-a84e-4709ce6a82eb-kube-api-access-rzcwp\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.210101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.210080    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qz9n\" (UniqueName: \"kubernetes.io/projected/c7c8981b-85d3-4d52-90b6-09d21f963de6-kube-api-access-8qz9n\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.210196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.210134    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkpz\" (UniqueName: \"kubernetes.io/projected/f649780e-1298-4f2b-a09f-b1527ee40c7c-kube-api-access-vvkpz\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.210352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.210334    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8pp\" (UniqueName: \"kubernetes.io/projected/d4067fb8-8775-4c03-a987-141bf11ca059-kube-api-access-wj8pp\") pod \"network-check-target-nr8hm\" (UID: \"d4067fb8-8775-4c03-a987-141bf11ca059\") " pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:28.213729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.213713    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5r99\" (UniqueName: \"kubernetes.io/projected/de991caa-9b7f-4a8a-8fad-e18008ef4e47-kube-api-access-t5r99\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.214165 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.214149    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4z8\" (UniqueName: \"kubernetes.io/projected/fc397c8b-8550-4340-a2db-91263d9fcec0-kube-api-access-2c4z8\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.215289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.215269    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tkm\" (UniqueName: \"kubernetes.io/projected/e282920c-4244-4ffd-87a8-a21d8c96d575-kube-api-access-67tkm\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.270624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.270598    6058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:28.270743 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.270726    6058 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:28.275971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.275951    6058 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:04:28.283737 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.283720    6058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:04:28.283835 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:28.283760    6058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:28.284916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.284896    6058 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:04:28.284992 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:28.284943    6058 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.us-east-2.compute.internal"
Oct 30 20:04:28.295241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295217    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:28.295334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295249    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.295334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295271    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-netns\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.295334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295295    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-etc-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.295334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295320    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5607f-deae-4832-b1ef-1ee7f095ca16-metrics-client-ca\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295343    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-socket-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295352    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-netns\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295370    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295386    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-etc-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295397    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-serving-certs-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295440    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-client-certs\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295458    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-socket-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295464    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-client-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295491    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295513    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-var-lib-kubelet\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.295548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295540    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-cert\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295556    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d414391-67f3-4f4b-aa36-022405704e89-metrics-certs\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295563    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295590    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295588    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5607f-deae-4832-b1ef-1ee7f095ca16-metrics-client-ca\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295648    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295648    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-serving-certs-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295808    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-cert\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295809    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295867    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-client-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295871    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-client-certs\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295887    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-var-lib-kubelet\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295905    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295931    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295946    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295964    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295990    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-federate-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296029    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296055    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-root\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.296069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296081    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz75\" (UniqueName: \"kubernetes.io/projected/d9f5607f-deae-4832-b1ef-1ee7f095ca16-kube-api-access-lpz75\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296084    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296105    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296142    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-conf-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296165    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-lib-modules\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296190    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmv5m\" (UniqueName: \"kubernetes.io/projected/4d414391-67f3-4f4b-aa36-022405704e89-kube-api-access-rmv5m\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296216    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296235    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296243    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-textfile\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296281    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296290    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-hostroot\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296348    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-hostroot\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296490    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-federate-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296533    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-systemd-units\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296566    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e282920c-4244-4ffd-87a8-a21d8c96d575-hosts-file\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296592    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbdx\" (UniqueName: \"kubernetes.io/projected/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-kube-api-access-srbdx\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296619    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296645    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d14bfe40-c6ab-48ff-a292-e67fc11dd857-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296675    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-system-cni-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296699    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-netd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296722    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-systemd\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296746    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296770    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5glt\" (UniqueName: \"kubernetes.io/projected/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-api-access-r5glt\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296794    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-modprobe-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296809    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296821    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-multus\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296791    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296901    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-conf-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296943    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-textfile\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.296969    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-modprobe-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297032    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-lib-modules\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297017    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e282920c-4244-4ffd-87a8-a21d8c96d575-hosts-file\") pod \"node-resolver-tdk9m\" (UID: \"e282920c-4244-4ffd-87a8-a21d8c96d575\") " pod="openshift-dns/node-resolver-tdk9m"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.295945    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297069    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297099    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297123    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-node-log\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297145    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6afece01-c595-4d76-8a37-9911ac9c9413-host-slash\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297165    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d14bfe40-c6ab-48ff-a292-e67fc11dd857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297169    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xfz\" (UniqueName: \"kubernetes.io/projected/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-kube-api-access-25xfz\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297266    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297315    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-systemd-units\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297067    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d14bfe40-c6ab-48ff-a292-e67fc11dd857-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297386    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-metrics-tls\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297399    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-node-log\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297420    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297467    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-socket-dir-parent\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297494    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-conf\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297501    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297516    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-multus\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297618    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-metrics-tls\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297657    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6afece01-c595-4d76-8a37-9911ac9c9413-host-slash\") pod \"iptables-alerter-fc6pd\" (UID: \"6afece01-c595-4d76-8a37-9911ac9c9413\") " pod="openshift-network-operator/iptables-alerter-fc6pd"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297681    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-systemd\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297708    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-socket-dir-parent\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297720    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-netd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297764    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297768    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-binary-copy\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297814    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297872    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297883    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297913    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-systemd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.297955    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298038    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-conf\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298052    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-tls\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.298029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298039    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-binary-copy\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298092    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysconfig\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298112    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298124    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cnibin\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298151    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298189    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjmj\" (UniqueName: \"kubernetes.io/projected/d14bfe40-c6ab-48ff-a292-e67fc11dd857-kube-api-access-qpjmj\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298222    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-wtmp\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298253    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-etc-kubernetes\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298278    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298294    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298310    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgrf\" (UniqueName: \"kubernetes.io/projected/a572a882-20a9-49fb-92cb-1f1d618ed9b0-kube-api-access-bfgrf\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298314    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298326    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf9894b4-5224-45ae-b205-86ad336937bb-audit-log\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298323    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-client-tls\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298343    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298088    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-systemd\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298357    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-tls\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298361    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc397c8b-8550-4340-a2db-91263d9fcec0-host\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298393    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298408    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-multus-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298411    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-netns\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298427    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf9894b4-5224-45ae-b205-86ad336937bb-audit-log\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298445    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-bin\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298475    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-cni-bin\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298487    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-netns\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298516    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298567    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysconfig\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298598    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-etc-kubernetes\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298604    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc397c8b-8550-4340-a2db-91263d9fcec0-host\") pod \"node-ca-nbkgn\" (UID: \"fc397c8b-8550-4340-a2db-91263d9fcec0\") " pod="openshift-image-registry/node-ca-nbkgn"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298620    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298637    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298645    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298667    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-metrics-server-audit-profiles\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298687    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298707    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298728    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-sys\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298752    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-metrics-client-ca\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298788    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298805    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298847    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-host\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298871    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298880    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jps\" (UniqueName: \"kubernetes.io/projected/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-kube-api-access-m9jps\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298911    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf9894b4-5224-45ae-b205-86ad336937bb-metrics-server-audit-profiles\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298913    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9edbdb-60a5-479c-a0de-c63c1725725e-crio-socket\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298950    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-tls\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298962    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-host\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298964    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.298991    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgcsk\" (UniqueName: \"kubernetes.io/projected/bf9894b4-5224-45ae-b205-86ad336937bb-kube-api-access-lgcsk\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299058    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299073    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a572a882-20a9-49fb-92cb-1f1d618ed9b0-metrics-client-ca\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299126    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-multus-certs\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299144    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-dbus\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299163    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299183    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299182    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-multus-certs\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299192    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf9894b4-5224-45ae-b205-86ad336937bb-secret-metrics-server-tls\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299237    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-slash\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299249    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-dbus\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299287    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-slash\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299296    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299350    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299395    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-device-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299488    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299514    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-kubelet\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299541    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-device-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299560    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-kubelet\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299564    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299596    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299593    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-run\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299565    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299623    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f5607f-deae-4832-b1ef-1ee7f095ca16-tmp\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299649    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-grpc-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299670    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-run\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299676    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-kubelet-config\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299697    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f5607f-deae-4832-b1ef-1ee7f095ca16-tmp\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299701    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-sys\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299725    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299749    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-bin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299756    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b57959a-5bcb-422a-bb6b-1d237cae31fb-kubelet-config\") pod \"global-pull-secret-syncer-sxthj\" (UID: \"2b57959a-5bcb-422a-bb6b-1d237cae31fb\") " pod="kube-system/global-pull-secret-syncer-sxthj"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299774    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-var-lib-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299798    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-log-socket\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299817    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-var-lib-cni-bin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299824    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299854    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtqh\" (UniqueName: \"kubernetes.io/projected/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-kube-api-access-9jtqh\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299868    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-sys\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299877    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-k8s-cni-cncf-io\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299915    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-log-socket\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299921    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299939    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-var-lib-openvswitch\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299955    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299960    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-kubernetes\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299975    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299996    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300048    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-metrics-client-ca\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300058    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-host-run-k8s-cni-cncf-io\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300020    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-kubernetes\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300074    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-registration-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300100    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-sys-fs\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300125    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-tmp-dir\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300152    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300163    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-ovn\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.299895    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-grpc-tls\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300198    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300239    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-sys-fs\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300255    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300289    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-kubelet\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300304    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300312    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300337    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300380    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9f5607f-deae-4832-b1ef-1ee7f095ca16-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300418    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300427    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-kubelet\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300448    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-run-ovn\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300419    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-metrics-client-ca\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300419    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-tmp-dir\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300501    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300506    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7c8981b-85d3-4d52-90b6-09d21f963de6-registration-dir\") pod \"aws-ebs-csi-driver-node-vnrb8\" (UID: \"c7c8981b-85d3-4d52-90b6-09d21f963de6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnrb8"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300524    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de991caa-9b7f-4a8a-8fad-e18008ef4e47-etc-sysctl-d\") pod \"tuned-pnfzm\" (UID: \"de991caa-9b7f-4a8a-8fad-e18008ef4e47\") " pod="openshift-cluster-node-tuning-operator/tuned-pnfzm"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300542    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300570    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300610    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-secret-telemeter-client\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300622    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020e2934-45d4-4998-a84e-4709ce6a82eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lscdq\" (UID: \"020e2934-45d4-4998-a84e-4709ce6a82eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300677    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300707    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300723    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300730    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-config-volume\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300762    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9edbdb-60a5-479c-a0de-c63c1725725e-data-volume\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300778    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9edbdb-60a5-479c-a0de-c63c1725725e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300793    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-cnibin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300817    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300842    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-system-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300863    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300887    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9edbdb-60a5-479c-a0de-c63c1725725e-data-volume\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300887    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300915    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-config-volume\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300923    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-system-cni-dir\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300930    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f649780e-1298-4f2b-a09f-b1527ee40c7c-cnibin\") pod \"multus-fbt2z\" (UID: \"f649780e-1298-4f2b-a09f-b1527ee40c7c\") " pod="openshift-multus/multus-fbt2z"
Oct 30 20:04:28.300965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.300986    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9edbdb-60a5-479c-a0de-c63c1725725e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.304618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.301152    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.312180 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.312155    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmv5m\" (UniqueName: \"kubernetes.io/projected/4d414391-67f3-4f4b-aa36-022405704e89-kube-api-access-rmv5m\") pod \"network-metrics-daemon-swsxv\" (UID: \"4d414391-67f3-4f4b-aa36-022405704e89\") " pod="openshift-multus/network-metrics-daemon-swsxv"
Oct 30 20:04:28.312690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.312672    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xfz\" (UniqueName: \"kubernetes.io/projected/e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9-kube-api-access-25xfz\") pod \"ingress-canary-z6nqg\" (UID: \"e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9\") " pod="openshift-ingress-canary/ingress-canary-z6nqg"
Oct 30 20:04:28.313116 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.313100    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz75\" (UniqueName: \"kubernetes.io/projected/d9f5607f-deae-4832-b1ef-1ee7f095ca16-kube-api-access-lpz75\") pod \"thanos-querier-6f7c4bc5db-qk2dh\" (UID: \"d9f5607f-deae-4832-b1ef-1ee7f095ca16\") " pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:28.313847 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.313826    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgrf\" (UniqueName: \"kubernetes.io/projected/a572a882-20a9-49fb-92cb-1f1d618ed9b0-kube-api-access-bfgrf\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.315022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.314980    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgcsk\" (UniqueName: \"kubernetes.io/projected/bf9894b4-5224-45ae-b205-86ad336937bb-kube-api-access-lgcsk\") pod \"metrics-server-64d548bdbc-6fsb4\" (UID: \"bf9894b4-5224-45ae-b205-86ad336937bb\") " pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:28.315088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.314983    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjmj\" (UniqueName: \"kubernetes.io/projected/d14bfe40-c6ab-48ff-a292-e67fc11dd857-kube-api-access-qpjmj\") pod \"prometheus-operator-9cd89b74c-9v99w\" (UID: \"d14bfe40-c6ab-48ff-a292-e67fc11dd857\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9v99w"
Oct 30 20:04:28.315175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.315147    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbdx\" (UniqueName: \"kubernetes.io/projected/c4caf5a9-050f-4f30-a76d-6f0bff4b4409-kube-api-access-srbdx\") pod \"dns-default-68nkf\" (UID: \"c4caf5a9-050f-4f30-a76d-6f0bff4b4409\") " pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:28.315271 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.315257    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") pod \"prometheus-k8s-0\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:28.315318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.315260    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5glt\" (UniqueName: \"kubernetes.io/projected/df9edbdb-60a5-479c-a0de-c63c1725725e-kube-api-access-r5glt\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.315698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.315676    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtqh\" (UniqueName: \"kubernetes.io/projected/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-kube-api-access-9jtqh\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.315971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.315954    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jps\" (UniqueName: \"kubernetes.io/projected/fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a-kube-api-access-m9jps\") pod \"telemeter-client-7564f9d8cc-bmrwv\" (UID: \"fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a\") " pod="openshift-monitoring/telemeter-client-7564f9d8cc-bmrwv"
Oct 30 20:04:28.402184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402134    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.402184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402175    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-root\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402197    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-system-cni-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402228    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cnibin\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402247    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-wtmp\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402247    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-root\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402265    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-sys\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402276    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-system-cni-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402246    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402299    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9edbdb-60a5-479c-a0de-c63c1725725e-crio-socket\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402324    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3cbde63-a1b5-4cd7-8700-1b9238fd7aad-cnibin\") pod \"multus-additional-cni-plugins-t42c6\" (UID: \"a3cbde63-a1b5-4cd7-8700-1b9238fd7aad\") " pod="openshift-multus/multus-additional-cni-plugins-t42c6"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402303    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-sys\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402343    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9edbdb-60a5-479c-a0de-c63c1725725e-crio-socket\") pod \"insights-runtime-extractor-rxgkl\" (UID: \"df9edbdb-60a5-479c-a0de-c63c1725725e\") " pod="openshift-insights/insights-runtime-extractor-rxgkl"
Oct 30 20:04:28.402373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:28.402343    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a572a882-20a9-49fb-92cb-1f1d618ed9b0-node-exporter-wtmp\") pod \"node-exporter-vccbk\" (UID: \"a572a882-20a9-49fb-92cb-1f1d618ed9b0\") " pod="openshift-monitoring/node-exporter-vccbk"
Oct 30 20:04:29.683295 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:29.683261    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-68nkf"
Oct 30 20:04:30.205479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:30.205452    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f7c4bc5db-qk2dh"
Oct 30 20:04:31.055226 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:31.055199    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:31.083239 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:31.083143    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:31.093448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:31.093427    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:04:31.100774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:31.100754    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:33.019436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:33.019402    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:04:33.855487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:33.855454    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:33.855696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:33.855666    6058 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:04:33.948045 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:33.947992    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lscdq"
Oct 30 20:04:49.642575 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:49.642542    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64d548bdbc-6fsb4"
Oct 30 20:04:49.654679 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:04:49.654648    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:04:53.217695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:04:53.217665    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ltkv6"
Oct 30 20:05:01.095974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:01.095939    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nr8hm"
Oct 30 20:05:19.653554 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:19.653518    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:05:33.035158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:33.035129    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:44.886222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886189    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:05:44.886785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886558    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="prometheus" containerID="cri-o://906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac" gracePeriod=600
Oct 30 20:05:44.886785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886600    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="thanos-sidecar" containerID="cri-o://dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311" gracePeriod=600
Oct 30 20:05:44.886785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886605    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy" containerID="cri-o://68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c" gracePeriod=600
Oct 30 20:05:44.886785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886675    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-web" containerID="cri-o://553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc" gracePeriod=600
Oct 30 20:05:44.886785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886735    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864" gracePeriod=600
Oct 30 20:05:44.886785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:44.886753    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="config-reloader" containerID="cri-o://26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35" gracePeriod=600
Oct 30 20:05:45.131138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.131107    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.229483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229402    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229440    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229471    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229488    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229515    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229545    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229580    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229609    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229634    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229678    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.229747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229721    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229756    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229786    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229797    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229819    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229856    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229888    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229922    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229954    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.229983    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230033    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230048    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") pod \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\" (UID: \"3ff12252-1ad0-44d1-8567-ca7ec029de1a\") "
Oct 30 20:05:45.230173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230098    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.230766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230304    6058 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.230766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230326    6058 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.230766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230344    6058 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-metrics-client-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.230766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230344    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.231067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.230823    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.231307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.231278    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.232640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.232611    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:45.233425 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.233398    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.233983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.233954    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:45.234894 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.234860    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.235127 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.235103    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "kube-api-access-zcvp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:45.235211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.235127    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.235529 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.235485    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.235625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.235588    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:45.236172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.236135    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.236254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.236200    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.236456 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.236431    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.236533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.236511    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.236651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.236634    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:45.244418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.244397    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config" pod "3ff12252-1ad0-44d1-8567-ca7ec029de1a" (UID: "3ff12252-1ad0-44d1-8567-ca7ec029de1a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:45.330749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330726    6058 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330749    6058 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-metrics-client-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330760    6058 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-kube-rbac-proxy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330770    6058 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330780    6058 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330789    6058 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-config-out\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330797    6058 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330806    6058 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330816    6058 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330824    6058 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330833    6058 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-db\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330842    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvp2\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-kube-api-access-zcvp2\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330852    6058 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.330863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330860    6058 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff12252-1ad0-44d1-8567-ca7ec029de1a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.331231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330870    6058 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-web-config\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.331231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330877    6058 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff12252-1ad0-44d1-8567-ca7ec029de1a-secret-grpc-tls\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.331231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.330886    6058 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff12252-1ad0-44d1-8567-ca7ec029de1a-tls-assets\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:45.517035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.516958    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864" exitCode=0
Oct 30 20:05:45.517035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.516978    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c" exitCode=0
Oct 30 20:05:45.517035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.516987    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc" exitCode=0
Oct 30 20:05:45.517035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.516992    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311" exitCode=0
Oct 30 20:05:45.517035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.516997    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35" exitCode=0
Oct 30 20:05:45.517035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517020    6058 generic.go:356] "Generic (PLEG): container finished" podID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac" exitCode=0
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517038    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"}
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517074    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517106    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"}
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517134    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"}
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517158    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"}
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517170    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517182    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"}
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517208    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"}
Oct 30 20:05:45.517378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.517231    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff12252-1ad0-44d1-8567-ca7ec029de1a","Type":"ContainerDied","Data":"610b64edc321a86b0e22069cb589a1715a2e088d31efb584334eea465b053b0e"}
Oct 30 20:05:45.524718 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.524664    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.532211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.532188    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.538431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.538413    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.542438 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.542418    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:05:45.545789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.545770    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.552030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.551986    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:05:45.552787 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.552771    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.559215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.559198    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.565163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565146    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.565429 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.565411    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.565475 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565436    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"} err="failed to get container status \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist"
Oct 30 20:05:45.565475 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565463    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.565698 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.565662    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.565738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565702    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"} err="failed to get container status \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist"
Oct 30 20:05:45.565738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565715    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.565916 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.565896    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.565957 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565920    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"} err="failed to get container status \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist"
Oct 30 20:05:45.565957 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.565932    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.566160 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.566143    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.566204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566164    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"} err="failed to get container status \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist"
Oct 30 20:05:45.566204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566176    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.566387 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.566373    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.566431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566390    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"} err="failed to get container status \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist"
Oct 30 20:05:45.566431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566401    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.566593 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.566577    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.566625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566596    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"} err="failed to get container status \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist"
Oct 30 20:05:45.566625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566614    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.566822 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:45.566807    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.566858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566825    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"} err="failed to get container status \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist"
Oct 30 20:05:45.566858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.566835    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.567034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567019    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"} err="failed to get container status \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist"
Oct 30 20:05:45.567078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567035    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.567259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567239    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"} err="failed to get container status \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist"
Oct 30 20:05:45.567298 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567261    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.567482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567460    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"} err="failed to get container status \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist"
Oct 30 20:05:45.567482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567476    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.567664 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567649    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"} err="failed to get container status \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist"
Oct 30 20:05:45.567703 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567664    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.567858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567844    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"} err="failed to get container status \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist"
Oct 30 20:05:45.567902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.567858    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.568051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568033    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"} err="failed to get container status \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist"
Oct 30 20:05:45.568051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568050    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.568228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568213    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"} err="failed to get container status \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist"
Oct 30 20:05:45.568263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568228    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.568405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568389    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"} err="failed to get container status \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist"
Oct 30 20:05:45.568445 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568405    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.568585 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568571    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"} err="failed to get container status \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist"
Oct 30 20:05:45.568630 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568585    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.568761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568747    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"} err="failed to get container status \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist"
Oct 30 20:05:45.568811 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568761    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.568953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568937    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"} err="failed to get container status \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist"
Oct 30 20:05:45.568991 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.568954    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.569157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569143    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"} err="failed to get container status \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist"
Oct 30 20:05:45.569204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569157    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.569338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569321    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"} err="failed to get container status \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist"
Oct 30 20:05:45.569376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569337    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.569515 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569500    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"} err="failed to get container status \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist"
Oct 30 20:05:45.569559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569515    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.569708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569686    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"} err="failed to get container status \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist"
Oct 30 20:05:45.569755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569708    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.569886 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569871    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"} err="failed to get container status \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist"
Oct 30 20:05:45.569927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.569885    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.570100 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570083    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"} err="failed to get container status \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist"
Oct 30 20:05:45.570151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570101    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.570311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570297    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"} err="failed to get container status \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist"
Oct 30 20:05:45.570352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570312    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.570503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570489    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"} err="failed to get container status \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist"
Oct 30 20:05:45.570544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570503    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.570687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570671    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"} err="failed to get container status \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist"
Oct 30 20:05:45.570726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570688    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.570858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570843    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"} err="failed to get container status \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist"
Oct 30 20:05:45.570895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.570857    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.571055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571040    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"} err="failed to get container status \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist"
Oct 30 20:05:45.571097 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571055    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.571247 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571232    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"} err="failed to get container status \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist"
Oct 30 20:05:45.571284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571247    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.571430 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571415    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"} err="failed to get container status \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist"
Oct 30 20:05:45.571474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571430    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.571613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571598    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"} err="failed to get container status \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist"
Oct 30 20:05:45.571652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571612    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.571779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571765    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"} err="failed to get container status \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist"
Oct 30 20:05:45.571842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571779    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.571957 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571942    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"} err="failed to get container status \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist"
Oct 30 20:05:45.571991 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.571957    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.572149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572134    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"} err="failed to get container status \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist"
Oct 30 20:05:45.572195 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572148    6058 scope.go:122] "RemoveContainer" containerID="fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"
Oct 30 20:05:45.572370 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572341    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864"} err="failed to get container status \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": rpc error: code = NotFound desc = could not find container \"fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864\": container with ID starting with fe7335cbac77e16de3e9afb99d2690685f1b26641ef8574e01f32c8eab28c864 not found: ID does not exist"
Oct 30 20:05:45.572370 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572366    6058 scope.go:122] "RemoveContainer" containerID="68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"
Oct 30 20:05:45.572547 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572528    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c"} err="failed to get container status \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": rpc error: code = NotFound desc = could not find container \"68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c\": container with ID starting with 68b7c65f3165a6c74b98d1c502ee444ea69b7ca7c1df799eb81158f8b6b1017c not found: ID does not exist"
Oct 30 20:05:45.572593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572548    6058 scope.go:122] "RemoveContainer" containerID="553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"
Oct 30 20:05:45.572725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572699    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc"} err="failed to get container status \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": rpc error: code = NotFound desc = could not find container \"553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc\": container with ID starting with 553000f881acc439d7b67bbe8a3e2c6cf432e23d2ed1fe0125aaa88ac9abfcbc not found: ID does not exist"
Oct 30 20:05:45.572725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572718    6058 scope.go:122] "RemoveContainer" containerID="dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"
Oct 30 20:05:45.572902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572883    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311"} err="failed to get container status \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": rpc error: code = NotFound desc = could not find container \"dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311\": container with ID starting with dd01e3c9194880326abc14f20416a5c34d3aaa2efd06b48c41c7b3154281b311 not found: ID does not exist"
Oct 30 20:05:45.572973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.572903    6058 scope.go:122] "RemoveContainer" containerID="26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"
Oct 30 20:05:45.573135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.573120    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35"} err="failed to get container status \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": rpc error: code = NotFound desc = could not find container \"26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35\": container with ID starting with 26b491fcf2b9d1f645e3390f9481ca424fa5bcfe5b314e50f8f183d84bb5ba35 not found: ID does not exist"
Oct 30 20:05:45.573135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.573135    6058 scope.go:122] "RemoveContainer" containerID="906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"
Oct 30 20:05:45.573327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.573309    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac"} err="failed to get container status \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": rpc error: code = NotFound desc = could not find container \"906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac\": container with ID starting with 906d0d16185fc8d2e8f9ae04b677d931822afe2054ec6f208d81e32ece731cac not found: ID does not exist"
Oct 30 20:05:45.573389 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.573328    6058 scope.go:122] "RemoveContainer" containerID="8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"
Oct 30 20:05:45.573528 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.573512    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2"} err="failed to get container status \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": rpc error: code = NotFound desc = could not find container \"8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2\": container with ID starting with 8fffdc901c7ab4e1f08df70701cb909837fea0923eb7cb939d38bac94c0aa1d2 not found: ID does not exist"
Oct 30 20:05:45.585098 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585069    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:05:45.585340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585327    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="config-reloader"
Oct 30 20:05:45.585384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585342    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="config-reloader"
Oct 30 20:05:45.585384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585360    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="init-config-reloader"
Oct 30 20:05:45.585384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585366    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="init-config-reloader"
Oct 30 20:05:45.585384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585373    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="thanos-sidecar"
Oct 30 20:05:45.585384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585378    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="thanos-sidecar"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585387    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="prometheus"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585392    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="prometheus"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585402    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585406    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585413    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-thanos"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585417    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-thanos"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585427    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-web"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585431    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-web"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585472    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="prometheus"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585480    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585487    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-thanos"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585494    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="kube-rbac-proxy-web"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585499    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="config-reloader"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585505    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="init-config-reloader"
Oct 30 20:05:45.585532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.585510    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" containerName="thanos-sidecar"
Oct 30 20:05:45.590529 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.590513    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.596318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596292    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:05:45.596423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596325    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:05:45.596486 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596433    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:05:45.596571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596552    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:05:45.596620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596578    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:05:45.596620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596601    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:05:45.596705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596650    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:05:45.596705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596658    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-84ke3oa62g28q\""
Oct 30 20:05:45.596864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596758    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:05:45.596864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596792    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:05:45.596864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596801    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:05:45.596864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596828    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:05:45.597051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.596928    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:05:45.597051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.597030    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f2fd8\""
Oct 30 20:05:45.602075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.602050    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:05:45.602888 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.602870    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:05:45.609949 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.609923    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:05:45.632698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632674    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632700    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632718    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632735    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632790    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632872    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632903    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.632945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.632924    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7v9j\" (UniqueName: \"kubernetes.io/projected/4451f2a7-0e2b-4764-a083-8ae831378bb2-kube-api-access-w7v9j\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633017    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633046    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633069    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633085    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4451f2a7-0e2b-4764-a083-8ae831378bb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633100    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633225 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633211    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633261 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633237    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633259    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633325 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633298    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633325 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633319    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-config\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633400 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633388    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4451f2a7-0e2b-4764-a083-8ae831378bb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.633431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.633408    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733643    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733675    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733707    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733736    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733759    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-config\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733795    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4451f2a7-0e2b-4764-a083-8ae831378bb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733819    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.733877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733849    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.733877    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734092    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734099    6058 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:05:45.734301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734119    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734163    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734192    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734224    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734252    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734276    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7v9j\" (UniqueName: \"kubernetes.io/projected/4451f2a7-0e2b-4764-a083-8ae831378bb2-kube-api-access-w7v9j\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734303    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734327    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734359    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734388    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4451f2a7-0e2b-4764-a083-8ae831378bb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734410    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734508    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734538    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.734921 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.734735    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.735314 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.735253    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.736036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.735989    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.736533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.736495    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.736937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.736911    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.737883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.737483    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4451f2a7-0e2b-4764-a083-8ae831378bb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.737883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.737652    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.737883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.737709    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.737883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.737842    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.738117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.737964    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.739316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.739291    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.739411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.739398    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-config\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.739471 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.739422    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4451f2a7-0e2b-4764-a083-8ae831378bb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.739471 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.739448    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.739532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.739493    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4451f2a7-0e2b-4764-a083-8ae831378bb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.739682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.739667    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4451f2a7-0e2b-4764-a083-8ae831378bb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.748065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.747982    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7v9j\" (UniqueName: \"kubernetes.io/projected/4451f2a7-0e2b-4764-a083-8ae831378bb2-kube-api-access-w7v9j\") pod \"prometheus-k8s-0\" (UID: \"4451f2a7-0e2b-4764-a083-8ae831378bb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:45.924368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:45.924124    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:05:46.221635 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:46.221528    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:05:46.224733 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:05:46.224689    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4451f2a7_0e2b_4764_a083_8ae831378bb2.slice/crio-7575600537a8e32334b47cb061521dfb01c4d12cc634e6647bbe892b5eb5bf85 WatchSource:0}: Error finding container 7575600537a8e32334b47cb061521dfb01c4d12cc634e6647bbe892b5eb5bf85: Status 404 returned error can't find the container with id 7575600537a8e32334b47cb061521dfb01c4d12cc634e6647bbe892b5eb5bf85
Oct 30 20:05:46.521241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:46.521171    6058 generic.go:356] "Generic (PLEG): container finished" podID="4451f2a7-0e2b-4764-a083-8ae831378bb2" containerID="3356c2134b0839ee68cea34f2050bed2e7951470ed22f6c903feb343f7458687" exitCode=0
Oct 30 20:05:46.521360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:46.521262    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerDied","Data":"3356c2134b0839ee68cea34f2050bed2e7951470ed22f6c903feb343f7458687"}
Oct 30 20:05:46.521360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:46.521338    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"7575600537a8e32334b47cb061521dfb01c4d12cc634e6647bbe892b5eb5bf85"}
Oct 30 20:05:47.214071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.212077    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff12252-1ad0-44d1-8567-ca7ec029de1a" path="/var/lib/kubelet/pods/3ff12252-1ad0-44d1-8567-ca7ec029de1a/volumes"
Oct 30 20:05:47.529143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.529063    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"ddfe83fe9459df70e486e82d6441f0a760df51748b100b7f4ba31a8e3ccab702"}
Oct 30 20:05:47.529264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.529145    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"9471efb60958dc0855da7f737c704b17b37d1fc2fd1fbf03e2348fb7220f8407"}
Oct 30 20:05:47.529264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.529186    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"636cdfa918e3cce627a95aee24e3819797ab8ee742b7637361e959ed6fb598cd"}
Oct 30 20:05:47.529264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.529226    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"75215ce4a2ac1b72aa8683c1801eb431baeaceba343532d796c2e9fb4a814b4e"}
Oct 30 20:05:47.529264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.529262    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"0d94df63806f88e7d11163cdd17e59cfb67ab9b565bbd5292256acb156914220"}
Oct 30 20:05:47.529415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.529300    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4451f2a7-0e2b-4764-a083-8ae831378bb2","Type":"ContainerStarted","Data":"d86e51b44f12215d9e9da1cc4262aa1e6b376bce2e05b56c468d9da29f39dfc3"}
Oct 30 20:05:47.567558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:47.567507    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.546023375 podStartE2EDuration="2.56749394s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.470565ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:47.565573926 +0000 UTC m=+80.468993953" watchObservedRunningTime="2026-10-30 20:05:47.56749394 +0000 UTC m=+80.470914005"
Oct 30 20:05:49.654728 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:05:49.654691    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:05:50.929186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:05:50.929148    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:06:19.653621 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:06:19.653541    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:06:45.929510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:06:45.929474    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:06:45.944413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:06:45.944385    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:06:46.737727 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:06:46.737699    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:06:49.654631 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:06:49.654589    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:07:19.654392 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:07:19.654340    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:07:49.653736 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:07:49.653694    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:08:19.654845 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:08:19.654809    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:08:37.128477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.128444    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"]
Oct 30 20:08:37.131596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.131570    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.141805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.141786    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:08:37.141805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.141796    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"]
Oct 30 20:08:37.141944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.141935    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 20:08:37.142472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.142459    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-m9kmr\""
Oct 30 20:08:37.231224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.231196    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9sh\" (UniqueName: \"kubernetes.io/projected/424d7fec-93de-4c32-95be-8c3f178e7c53-kube-api-access-4x9sh\") pod \"openshift-lws-operator-8465994787-ttvkv\" (UID: \"424d7fec-93de-4c32-95be-8c3f178e7c53\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.231350 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.231249    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/424d7fec-93de-4c32-95be-8c3f178e7c53-tmp\") pod \"openshift-lws-operator-8465994787-ttvkv\" (UID: \"424d7fec-93de-4c32-95be-8c3f178e7c53\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.332573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.332544    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9sh\" (UniqueName: \"kubernetes.io/projected/424d7fec-93de-4c32-95be-8c3f178e7c53-kube-api-access-4x9sh\") pod \"openshift-lws-operator-8465994787-ttvkv\" (UID: \"424d7fec-93de-4c32-95be-8c3f178e7c53\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.332702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.332625    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/424d7fec-93de-4c32-95be-8c3f178e7c53-tmp\") pod \"openshift-lws-operator-8465994787-ttvkv\" (UID: \"424d7fec-93de-4c32-95be-8c3f178e7c53\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.333021 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.332988    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/424d7fec-93de-4c32-95be-8c3f178e7c53-tmp\") pod \"openshift-lws-operator-8465994787-ttvkv\" (UID: \"424d7fec-93de-4c32-95be-8c3f178e7c53\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.345922 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.345897    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9sh\" (UniqueName: \"kubernetes.io/projected/424d7fec-93de-4c32-95be-8c3f178e7c53-kube-api-access-4x9sh\") pod \"openshift-lws-operator-8465994787-ttvkv\" (UID: \"424d7fec-93de-4c32-95be-8c3f178e7c53\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.460390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.460336    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"
Oct 30 20:08:37.730439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.730361    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv"]
Oct 30 20:08:37.733860 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:08:37.733834    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod424d7fec_93de_4c32_95be_8c3f178e7c53.slice/crio-48737a007e7a1f8ca30cfef7f22c341fd3b0b11ad9f714d7833fe01f9dccbc25 WatchSource:0}: Error finding container 48737a007e7a1f8ca30cfef7f22c341fd3b0b11ad9f714d7833fe01f9dccbc25: Status 404 returned error can't find the container with id 48737a007e7a1f8ca30cfef7f22c341fd3b0b11ad9f714d7833fe01f9dccbc25
Oct 30 20:08:37.735357 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:37.735338    6058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:08:38.081778 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:38.081700    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv" event={"ID":"424d7fec-93de-4c32-95be-8c3f178e7c53","Type":"ContainerStarted","Data":"48737a007e7a1f8ca30cfef7f22c341fd3b0b11ad9f714d7833fe01f9dccbc25"}
Oct 30 20:08:41.097038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:41.096989    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv" event={"ID":"424d7fec-93de-4c32-95be-8c3f178e7c53","Type":"ContainerStarted","Data":"2c253d64ad7e902c79f3f81b4879bf45c50f29f176521b7eb9fdb4e5491b600a"}
Oct 30 20:08:41.123029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:41.118762    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ttvkv" podStartSLOduration=0.925732222 podStartE2EDuration="4.118746078s" totalImagesPullingTime="3.193013856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:41.117151239 +0000 UTC m=+254.020571265" watchObservedRunningTime="2026-10-30 20:08:41.118746078 +0000 UTC m=+254.022166106"
Oct 30 20:08:49.654299 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:08:49.654257    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:08:55.506689 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.506654    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"]
Oct 30 20:08:55.512176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.512156    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.548086 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.548063    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"]
Oct 30 20:08:55.568257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.568230    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:08:55.568380 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.568365    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:08:55.587699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.587681    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:08:55.587935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.587918    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:08:55.587973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.587962    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:08:55.588109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.588097    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d9p54\""
Oct 30 20:08:55.657658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.657635    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef85828-b3d6-4b6f-b0be-a061b570503f-tmp\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.657766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.657668    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.657766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.657707    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.657846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.657775    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ef85828-b3d6-4b6f-b0be-a061b570503f-manifests\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.657846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.657796    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpv4\" (UniqueName: \"kubernetes.io/projected/9ef85828-b3d6-4b6f-b0be-a061b570503f-kube-api-access-swpv4\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.657846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.657829    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758266    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef85828-b3d6-4b6f-b0be-a061b570503f-tmp\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758301    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758338    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758367    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ef85828-b3d6-4b6f-b0be-a061b570503f-manifests\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758395    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpv4\" (UniqueName: \"kubernetes.io/projected/9ef85828-b3d6-4b6f-b0be-a061b570503f-kube-api-access-swpv4\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758443    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.758825 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.758797    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ef85828-b3d6-4b6f-b0be-a061b570503f-manifests\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.760883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.760861    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.760961 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.760909    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.761036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.760997    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef85828-b3d6-4b6f-b0be-a061b570503f-tmp\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.761156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.761140    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ef85828-b3d6-4b6f-b0be-a061b570503f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.779771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.779753    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpv4\" (UniqueName: \"kubernetes.io/projected/9ef85828-b3d6-4b6f-b0be-a061b570503f-kube-api-access-swpv4\") pod \"opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp\" (UID: \"9ef85828-b3d6-4b6f-b0be-a061b570503f\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:55.826626 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:55.826602    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:08:56.152176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:56.152154    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"]
Oct 30 20:08:56.154872 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:08:56.154841    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef85828_b3d6_4b6f_b0be_a061b570503f.slice/crio-68162240ce3ed2beeb046dc5e27ae11404f3e10f3fd1a511f5d1df89a930f23b WatchSource:0}: Error finding container 68162240ce3ed2beeb046dc5e27ae11404f3e10f3fd1a511f5d1df89a930f23b: Status 404 returned error can't find the container with id 68162240ce3ed2beeb046dc5e27ae11404f3e10f3fd1a511f5d1df89a930f23b
Oct 30 20:08:57.151924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:57.151844    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp" event={"ID":"9ef85828-b3d6-4b6f-b0be-a061b570503f","Type":"ContainerStarted","Data":"68162240ce3ed2beeb046dc5e27ae11404f3e10f3fd1a511f5d1df89a930f23b"}
Oct 30 20:08:59.161293 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:08:59.161265    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp" event={"ID":"9ef85828-b3d6-4b6f-b0be-a061b570503f","Type":"ContainerStarted","Data":"27074d93f678efb38dcc2b1e5d66462cd8edc5fdd7915ca5b08a9de4542678fc"}
Oct 30 20:09:00.166429 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:00.166392    6058 generic.go:356] "Generic (PLEG): container finished" podID="9ef85828-b3d6-4b6f-b0be-a061b570503f" containerID="27074d93f678efb38dcc2b1e5d66462cd8edc5fdd7915ca5b08a9de4542678fc" exitCode=0
Oct 30 20:09:00.166803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:00.166476    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp" event={"ID":"9ef85828-b3d6-4b6f-b0be-a061b570503f","Type":"ContainerDied","Data":"27074d93f678efb38dcc2b1e5d66462cd8edc5fdd7915ca5b08a9de4542678fc"}
Oct 30 20:09:01.171342 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:01.171301    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp" event={"ID":"9ef85828-b3d6-4b6f-b0be-a061b570503f","Type":"ContainerStarted","Data":"6d491da53a337406ef22d131fede950b9e27199be7cbd18f0aeec2d3f057d763"}
Oct 30 20:09:01.171727 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:01.171428    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:09:01.219938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:01.219889    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp" podStartSLOduration=2.524799042 podStartE2EDuration="6.219876392s" totalImagesPullingTime="3.204671381s" totalInitContainerRuntime="490.405969ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:01.217890004 +0000 UTC m=+274.121310032" watchObservedRunningTime="2026-10-30 20:09:01.219876392 +0000 UTC m=+274.123296415"
Oct 30 20:09:12.177780 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:12.177695    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp"
Oct 30 20:09:16.656051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.655996    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"]
Oct 30 20:09:16.662451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.662401    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.671929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.671901    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"]
Oct 30 20:09:16.672386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.672369    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 20:09:16.672458 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.672446    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 20:09:16.672506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.672495    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-h9wrx\""
Oct 30 20:09:16.672554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.672545    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 20:09:16.711718 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.711693    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81c1509d-18cd-4b10-895c-43c667f4cff0-cert\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.711844 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.711732    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njzm\" (UniqueName: \"kubernetes.io/projected/81c1509d-18cd-4b10-895c-43c667f4cff0-kube-api-access-6njzm\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.711844 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.711769    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81c1509d-18cd-4b10-895c-43c667f4cff0-metrics-cert\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.711918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.711893    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81c1509d-18cd-4b10-895c-43c667f4cff0-manager-config\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.813110 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.813062    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81c1509d-18cd-4b10-895c-43c667f4cff0-metrics-cert\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.813307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.813127    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81c1509d-18cd-4b10-895c-43c667f4cff0-manager-config\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.813307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.813153    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81c1509d-18cd-4b10-895c-43c667f4cff0-cert\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.813307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.813172    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njzm\" (UniqueName: \"kubernetes.io/projected/81c1509d-18cd-4b10-895c-43c667f4cff0-kube-api-access-6njzm\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.813877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.813856    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81c1509d-18cd-4b10-895c-43c667f4cff0-manager-config\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.815391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.815366    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81c1509d-18cd-4b10-895c-43c667f4cff0-cert\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.815518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.815436    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81c1509d-18cd-4b10-895c-43c667f4cff0-metrics-cert\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.826318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.826294    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njzm\" (UniqueName: \"kubernetes.io/projected/81c1509d-18cd-4b10-895c-43c667f4cff0-kube-api-access-6njzm\") pod \"lws-controller-manager-8656d6df85-bbps4\" (UID: \"81c1509d-18cd-4b10-895c-43c667f4cff0\") " pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:16.974958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:16.974897    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:17.246483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:17.246412    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"]
Oct 30 20:09:17.250241 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:09:17.250213    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c1509d_18cd_4b10_895c_43c667f4cff0.slice/crio-ce1a6b0925fab2f86d2939dbd36891fca9241566a97e660de410221a720a3f54 WatchSource:0}: Error finding container ce1a6b0925fab2f86d2939dbd36891fca9241566a97e660de410221a720a3f54: Status 404 returned error can't find the container with id ce1a6b0925fab2f86d2939dbd36891fca9241566a97e660de410221a720a3f54
Oct 30 20:09:18.237289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:18.237251    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4" event={"ID":"81c1509d-18cd-4b10-895c-43c667f4cff0","Type":"ContainerStarted","Data":"ce1a6b0925fab2f86d2939dbd36891fca9241566a97e660de410221a720a3f54"}
Oct 30 20:09:19.652646 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:09:19.652602    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:09:20.245285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:20.245251    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4" event={"ID":"81c1509d-18cd-4b10-895c-43c667f4cff0","Type":"ContainerStarted","Data":"7df9bfd85711eae53936be9b8c36b345122c30c770e3eaba486779d7b6be4ac4"}
Oct 30 20:09:20.245439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:20.245415    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:20.266184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:20.266134    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4" podStartSLOduration=2.041307419 podStartE2EDuration="4.266123199s" totalImagesPullingTime="2.22481578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:20.265452743 +0000 UTC m=+293.168872769" watchObservedRunningTime="2026-10-30 20:09:20.266123199 +0000 UTC m=+293.169543225"
Oct 30 20:09:25.170473 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.170441    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"]
Oct 30 20:09:25.173913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.173888    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.183726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.183699    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:09:25.184624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.184606    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:09:25.192285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.192255    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"]
Oct 30 20:09:25.227890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.227857    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:09:25.227890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.227868    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 20:09:25.228063 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.227927    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9dr5w\""
Oct 30 20:09:25.275718 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.275692    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-tmp\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.275952 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.275932    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-tls-certs\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.276070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.275967    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5gq\" (UniqueName: \"kubernetes.io/projected/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-kube-api-access-fs5gq\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.377101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.377073    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-tls-certs\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.377217 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.377102    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5gq\" (UniqueName: \"kubernetes.io/projected/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-kube-api-access-fs5gq\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.377217 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.377139    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-tmp\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.379376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.379349    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-tmp\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.379512 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.379498    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-tls-certs\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.389599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.389578    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5gq\" (UniqueName: \"kubernetes.io/projected/d3ec1597-b02b-469a-b2b8-7c1a085b76f9-kube-api-access-fs5gq\") pod \"kube-auth-proxy-d477969dd-rzcf7\" (UID: \"d3ec1597-b02b-469a-b2b8-7c1a085b76f9\") " pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.485677 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.485631    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"
Oct 30 20:09:25.764228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:25.764158    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d477969dd-rzcf7"]
Oct 30 20:09:25.767942 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:09:25.767917    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ec1597_b02b_469a_b2b8_7c1a085b76f9.slice/crio-45f75a0c718f429c882f174f697d880579e58ea49b6ad4566fa0b0d9c1d7e230 WatchSource:0}: Error finding container 45f75a0c718f429c882f174f697d880579e58ea49b6ad4566fa0b0d9c1d7e230: Status 404 returned error can't find the container with id 45f75a0c718f429c882f174f697d880579e58ea49b6ad4566fa0b0d9c1d7e230
Oct 30 20:09:26.266048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:26.266015    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7" event={"ID":"d3ec1597-b02b-469a-b2b8-7c1a085b76f9","Type":"ContainerStarted","Data":"45f75a0c718f429c882f174f697d880579e58ea49b6ad4566fa0b0d9c1d7e230"}
Oct 30 20:09:28.089203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:28.089175    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:09:28.275398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:28.275360    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7" event={"ID":"d3ec1597-b02b-469a-b2b8-7c1a085b76f9","Type":"ContainerStarted","Data":"60202da4b32c3e9b69a054f7fbe3e62d66a5c66e5e4604ac67125138926d75b8"}
Oct 30 20:09:28.295968 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:28.295917    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d477969dd-rzcf7" podStartSLOduration=0.979718012 podStartE2EDuration="3.295903913s" totalImagesPullingTime="2.316185901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:28.294096242 +0000 UTC m=+301.197516280" watchObservedRunningTime="2026-10-30 20:09:28.295903913 +0000 UTC m=+301.199323938"
Oct 30 20:09:31.251487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:09:31.251457    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8656d6df85-bbps4"
Oct 30 20:09:49.654691 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:09:49.654656    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:10:19.654236 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:10:19.654199    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:10:49.654168 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:10:49.654132    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:11:19.653474 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:19.653433    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:11:33.592194 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.592158    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"]
Oct 30 20:11:33.595569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.595542    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:33.600606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.600586    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 20:11:33.612070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.612040    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:11:33.612070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.612055    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:11:33.612266 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.612234    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qzvnj\""
Oct 30 20:11:33.613001 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.612980    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"]
Oct 30 20:11:33.733812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.733784    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkvxj\" (UniqueName: \"kubernetes.io/projected/bc8a5270-0a47-47db-9d1a-1ec0dad81ce5-kube-api-access-qkvxj\") pod \"dns-operator-controller-manager-65bfd79598-mdr4m\" (UID: \"bc8a5270-0a47-47db-9d1a-1ec0dad81ce5\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:33.834474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.834450    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkvxj\" (UniqueName: \"kubernetes.io/projected/bc8a5270-0a47-47db-9d1a-1ec0dad81ce5-kube-api-access-qkvxj\") pod \"dns-operator-controller-manager-65bfd79598-mdr4m\" (UID: \"bc8a5270-0a47-47db-9d1a-1ec0dad81ce5\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:33.849128 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.849074    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkvxj\" (UniqueName: \"kubernetes.io/projected/bc8a5270-0a47-47db-9d1a-1ec0dad81ce5-kube-api-access-qkvxj\") pod \"dns-operator-controller-manager-65bfd79598-mdr4m\" (UID: \"bc8a5270-0a47-47db-9d1a-1ec0dad81ce5\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:33.908542 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:33.908518    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:34.179272 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:34.179239    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"]
Oct 30 20:11:34.182802 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:11:34.182773    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8a5270_0a47_47db_9d1a_1ec0dad81ce5.slice/crio-4373f3a463d109b77813aef3d3fc34002a8822ba878517212d784b9bc4d8df8d WatchSource:0}: Error finding container 4373f3a463d109b77813aef3d3fc34002a8822ba878517212d784b9bc4d8df8d: Status 404 returned error can't find the container with id 4373f3a463d109b77813aef3d3fc34002a8822ba878517212d784b9bc4d8df8d
Oct 30 20:11:34.713038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:34.712989    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m" event={"ID":"bc8a5270-0a47-47db-9d1a-1ec0dad81ce5","Type":"ContainerStarted","Data":"4373f3a463d109b77813aef3d3fc34002a8822ba878517212d784b9bc4d8df8d"}
Oct 30 20:11:38.384779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.384747    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"]
Oct 30 20:11:38.387902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.387885    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.393501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.393480    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t8dtz\""
Oct 30 20:11:38.402740 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.402718    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"]
Oct 30 20:11:38.569235 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.569207    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.569350 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.569260    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wv8v\" (UniqueName: \"kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.670422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.670355    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wv8v\" (UniqueName: \"kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.670422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.670410    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.670728 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.670696    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.683741 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.683717    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wv8v\" (UniqueName: \"kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.700355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.700333    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:38.990197 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:38.990071    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"]
Oct 30 20:11:38.995475 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:11:38.995435    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2012dc_9acb_4320_a640_62a4dcacf786.slice/crio-40d3a8baccbf8ee2810de3f2411fa8a60318764f95b7a5d38d911798ee2dd898 WatchSource:0}: Error finding container 40d3a8baccbf8ee2810de3f2411fa8a60318764f95b7a5d38d911798ee2dd898: Status 404 returned error can't find the container with id 40d3a8baccbf8ee2810de3f2411fa8a60318764f95b7a5d38d911798ee2dd898
Oct 30 20:11:39.734539 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:39.734503    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l" event={"ID":"fa2012dc-9acb-4320-a640-62a4dcacf786","Type":"ContainerStarted","Data":"40d3a8baccbf8ee2810de3f2411fa8a60318764f95b7a5d38d911798ee2dd898"}
Oct 30 20:11:40.740831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:40.740784    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m" event={"ID":"bc8a5270-0a47-47db-9d1a-1ec0dad81ce5","Type":"ContainerStarted","Data":"cee0f7c3e4c51ca3bde3a3ebd54c241e873cbfb632180c7611bdd5085cf5743b"}
Oct 30 20:11:40.741238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:40.741053    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:40.763499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:40.763438    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m" podStartSLOduration=1.337068398 podStartE2EDuration="7.763420104s" totalImagesPullingTime="6.426351706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:40.762577581 +0000 UTC m=+433.665997608" watchObservedRunningTime="2026-10-30 20:11:40.763420104 +0000 UTC m=+433.666840132"
Oct 30 20:11:44.759176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:44.759129    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l" event={"ID":"fa2012dc-9acb-4320-a640-62a4dcacf786","Type":"ContainerStarted","Data":"d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420"}
Oct 30 20:11:44.759565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:44.759223    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:44.783486 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:44.783436    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l" podStartSLOduration=1.813728957 podStartE2EDuration="6.783424008s" totalImagesPullingTime="4.969695051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:44.781933564 +0000 UTC m=+437.685353587" watchObservedRunningTime="2026-10-30 20:11:44.783424008 +0000 UTC m=+437.686844034"
Oct 30 20:11:49.654486 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:49.654442    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:11:51.747778 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:51.747751    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-mdr4m"
Oct 30 20:11:55.765562 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:55.765532    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.494751 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.494710    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"]
Oct 30 20:11:59.495247 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.494930    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" containerName="manager" containerID="cri-o://d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420" gracePeriod=2
Oct 30 20:11:59.503946 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.503915    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"]
Oct 30 20:11:59.513624 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:59.513581    6058 status_manager.go:1068] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa2012dc-9acb-4320-a640-62a4dcacf786\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-6b6fb56846-c526l\": pods \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.528363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.528336    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"]
Oct 30 20:11:59.528686 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.528673    6058 cpu_manager.go:414] "containerMap: removing container" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" containerName="manager"
Oct 30 20:11:59.528734 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.528690    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" containerName="manager"
Oct 30 20:11:59.528771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.528764    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" containerName="manager"
Oct 30 20:11:59.532234 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.532211    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.534938 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:59.534907    6058 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" is forbidden: User \"system:node:ip-10-0-141-189.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-189.us-east-2.compute.internal' and this object" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.547492 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.547467    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"]
Oct 30 20:11:59.628451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.628422    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p\") pod \"kuadrant-operator-controller-manager-6b6fb56846-r2sxm\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.628610 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.628457    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-r2sxm\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.729032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.728982    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p\") pod \"kuadrant-operator-controller-manager-6b6fb56846-r2sxm\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.729153 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.729046    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-r2sxm\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.729455 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.729422    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-r2sxm\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.733104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.733082    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.735577 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:59.735554    6058 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" is forbidden: User \"system:node:ip-10-0-141-189.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-189.us-east-2.compute.internal' and this object" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.741876 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.741855    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p\") pod \"kuadrant-operator-controller-manager-6b6fb56846-r2sxm\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.815983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.815926    6058 generic.go:356] "Generic (PLEG): container finished" podID="fa2012dc-9acb-4320-a640-62a4dcacf786" containerID="d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420" exitCode=0
Oct 30 20:11:59.815983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.815973    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.816142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.816045    6058 scope.go:122] "RemoveContainer" containerID="d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420"
Oct 30 20:11:59.818697 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:59.818670    6058 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" is forbidden: User \"system:node:ip-10-0-141-189.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-189.us-east-2.compute.internal' and this object" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:11:59.824343 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.824325    6058 scope.go:122] "RemoveContainer" containerID="d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420"
Oct 30 20:11:59.824618 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:11:59.824599    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420\": container with ID starting with d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420 not found: ID does not exist" containerID="d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420"
Oct 30 20:11:59.824680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.824638    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420"} err="failed to get container status \"d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420\": rpc error: code = NotFound desc = could not find container \"d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420\": container with ID starting with d92d0641085ceac6ec2b4eced4a60f81dbbdc2d21d6c23c7ad747f5820d7f420 not found: ID does not exist"
Oct 30 20:11:59.829840 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.829822    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v\" (UniqueName: \"kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v\") pod \"fa2012dc-9acb-4320-a640-62a4dcacf786\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") "
Oct 30 20:11:59.829934 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.829923    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume\") pod \"fa2012dc-9acb-4320-a640-62a4dcacf786\" (UID: \"fa2012dc-9acb-4320-a640-62a4dcacf786\") "
Oct 30 20:11:59.830399 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.830380    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume" pod "fa2012dc-9acb-4320-a640-62a4dcacf786" (UID: "fa2012dc-9acb-4320-a640-62a4dcacf786"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:59.831724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.831698    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v" pod "fa2012dc-9acb-4320-a640-62a4dcacf786" (UID: "fa2012dc-9acb-4320-a640-62a4dcacf786"). InnerVolumeSpecName "kube-api-access-6wv8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:59.902495 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.902469    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:11:59.930807 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.930777    6058 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa2012dc-9acb-4320-a640-62a4dcacf786-extensions-socket-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:59.930897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:11:59.930824    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wv8v\" (UniqueName: \"kubernetes.io/projected/fa2012dc-9acb-4320-a640-62a4dcacf786-kube-api-access-6wv8v\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:00.127527 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:12:00.127496    6058 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" is forbidden: User \"system:node:ip-10-0-141-189.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-189.us-east-2.compute.internal' and this object" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:12:00.192420 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:00.192397    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"]
Oct 30 20:12:00.194542 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:12:00.194512    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd73bad_e9ce_4101_8e9e_22e0c1690a64.slice/crio-5efa10e0918cfc534efac362139bb05f50765d4aca53ee3652fb3020469703d9 WatchSource:0}: Error finding container 5efa10e0918cfc534efac362139bb05f50765d4aca53ee3652fb3020469703d9: Status 404 returned error can't find the container with id 5efa10e0918cfc534efac362139bb05f50765d4aca53ee3652fb3020469703d9
Oct 30 20:12:00.822174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:00.822138    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm" event={"ID":"5dd73bad-e9ce-4101-8e9e-22e0c1690a64","Type":"ContainerStarted","Data":"acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7"}
Oct 30 20:12:00.822628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:00.822198    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm" event={"ID":"5dd73bad-e9ce-4101-8e9e-22e0c1690a64","Type":"ContainerStarted","Data":"5efa10e0918cfc534efac362139bb05f50765d4aca53ee3652fb3020469703d9"}
Oct 30 20:12:00.822628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:00.822226    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:12:00.824831 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:12:00.824805    6058 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-c526l\" is forbidden: User \"system:node:ip-10-0-141-189.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-189.us-east-2.compute.internal' and this object" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c526l"
Oct 30 20:12:00.851492 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:00.851431    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm" podStartSLOduration=1.851417916 podStartE2EDuration="1.851417916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:00.849457844 +0000 UTC m=+453.752877869" watchObservedRunningTime="2026-10-30 20:12:00.851417916 +0000 UTC m=+453.754837942"
Oct 30 20:12:01.210728 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:01.210697    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2012dc-9acb-4320-a640-62a4dcacf786" path="/var/lib/kubelet/pods/fa2012dc-9acb-4320-a640-62a4dcacf786/volumes"
Oct 30 20:12:11.829265 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:11.829197    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:12:14.129883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.129850    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"]
Oct 30 20:12:14.130290 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.130060    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm" podUID="5dd73bad-e9ce-4101-8e9e-22e0c1690a64" containerName="manager" containerID="cri-o://acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7" gracePeriod=10
Oct 30 20:12:14.372073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.372047    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:12:14.429228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.429141    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"]
Oct 30 20:12:14.429628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.429610    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5dd73bad-e9ce-4101-8e9e-22e0c1690a64" containerName="manager"
Oct 30 20:12:14.429670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.429630    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd73bad-e9ce-4101-8e9e-22e0c1690a64" containerName="manager"
Oct 30 20:12:14.429727 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.429715    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd73bad-e9ce-4101-8e9e-22e0c1690a64" containerName="manager"
Oct 30 20:12:14.432573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.432557    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.445255 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.445237    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume\") pod \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") "
Oct 30 20:12:14.445348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.445300    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p\") pod \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\" (UID: \"5dd73bad-e9ce-4101-8e9e-22e0c1690a64\") "
Oct 30 20:12:14.445625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.445604    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume" pod "5dd73bad-e9ce-4101-8e9e-22e0c1690a64" (UID: "5dd73bad-e9ce-4101-8e9e-22e0c1690a64"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.447232 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.447210    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p" pod "5dd73bad-e9ce-4101-8e9e-22e0c1690a64" (UID: "5dd73bad-e9ce-4101-8e9e-22e0c1690a64"). InnerVolumeSpecName "kube-api-access-v4k8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.464831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.464801    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"]
Oct 30 20:12:14.546347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.546319    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-s2x2p\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.546469 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.546390    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf8d\" (UniqueName: \"kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d\") pod \"kuadrant-operator-controller-manager-57bb9bfff-s2x2p\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.546511 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.546463    6058 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-extensions-socket-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.546511 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.546482    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/5dd73bad-e9ce-4101-8e9e-22e0c1690a64-kube-api-access-v4k8p\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.646920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.646890    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf8d\" (UniqueName: \"kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d\") pod \"kuadrant-operator-controller-manager-57bb9bfff-s2x2p\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.647041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.646929    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-s2x2p\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.647327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.647310    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-s2x2p\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.659477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.659454    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf8d\" (UniqueName: \"kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d\") pod \"kuadrant-operator-controller-manager-57bb9bfff-s2x2p\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.743863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.743813    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:14.878180 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.878148    6058 generic.go:356] "Generic (PLEG): container finished" podID="5dd73bad-e9ce-4101-8e9e-22e0c1690a64" containerID="acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7" exitCode=0
Oct 30 20:12:14.878332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.878197    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"
Oct 30 20:12:14.878332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.878234    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm" event={"ID":"5dd73bad-e9ce-4101-8e9e-22e0c1690a64","Type":"ContainerDied","Data":"acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7"}
Oct 30 20:12:14.878332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.878291    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm" event={"ID":"5dd73bad-e9ce-4101-8e9e-22e0c1690a64","Type":"ContainerDied","Data":"5efa10e0918cfc534efac362139bb05f50765d4aca53ee3652fb3020469703d9"}
Oct 30 20:12:14.878332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.878314    6058 scope.go:122] "RemoveContainer" containerID="acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7"
Oct 30 20:12:14.887257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.887237    6058 scope.go:122] "RemoveContainer" containerID="acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7"
Oct 30 20:12:14.887501 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:12:14.887484    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7\": container with ID starting with acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7 not found: ID does not exist" containerID="acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7"
Oct 30 20:12:14.887555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.887511    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7"} err="failed to get container status \"acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7\": rpc error: code = NotFound desc = could not find container \"acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7\": container with ID starting with acaa39def6ab104c22acce001e4387d22813178d861df74c2b2c18306ca47db7 not found: ID does not exist"
Oct 30 20:12:14.908594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.908574    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"]
Oct 30 20:12:14.915215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:14.915192    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-r2sxm"]
Oct 30 20:12:15.031688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:15.031592    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"]
Oct 30 20:12:15.034848 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:12:15.034817    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae638cf2_46c5_4c5c_a383_ae889399a8a6.slice/crio-a6c7e107b381f8b4009f3f272fd4554420141c9e43f3284828a6d35261a275dd WatchSource:0}: Error finding container a6c7e107b381f8b4009f3f272fd4554420141c9e43f3284828a6d35261a275dd: Status 404 returned error can't find the container with id a6c7e107b381f8b4009f3f272fd4554420141c9e43f3284828a6d35261a275dd
Oct 30 20:12:15.210818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:15.210789    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd73bad-e9ce-4101-8e9e-22e0c1690a64" path="/var/lib/kubelet/pods/5dd73bad-e9ce-4101-8e9e-22e0c1690a64/volumes"
Oct 30 20:12:15.886769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:15.886718    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p" event={"ID":"ae638cf2-46c5-4c5c-a383-ae889399a8a6","Type":"ContainerStarted","Data":"2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4"}
Oct 30 20:12:15.886973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:15.886792    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p" event={"ID":"ae638cf2-46c5-4c5c-a383-ae889399a8a6","Type":"ContainerStarted","Data":"a6c7e107b381f8b4009f3f272fd4554420141c9e43f3284828a6d35261a275dd"}
Oct 30 20:12:15.886973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:15.886826    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:15.913558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:15.913512    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p" podStartSLOduration=1.91349671 podStartE2EDuration="1.91349671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:15.911941667 +0000 UTC m=+468.815361693" watchObservedRunningTime="2026-10-30 20:12:15.91349671 +0000 UTC m=+468.816916738"
Oct 30 20:12:19.653554 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:12:19.653514    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:12:26.894678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:26.894648    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:12:42.461060 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.461027    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:12:42.464414 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.464392    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.474335 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.474316    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6qhrp\""
Oct 30 20:12:42.484125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.484106    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:12:42.499590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.499572    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 20:12:42.525851 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.525829    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:12:42.553344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.553321    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a35d36de-bc9b-4740-b19e-9efb37df2d2f-config-file\") pod \"limitador-limitador-7c9b87d574-h2vml\" (UID: \"a35d36de-bc9b-4740-b19e-9efb37df2d2f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.553429 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.553369    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22p4l\" (UniqueName: \"kubernetes.io/projected/a35d36de-bc9b-4740-b19e-9efb37df2d2f-kube-api-access-22p4l\") pod \"limitador-limitador-7c9b87d574-h2vml\" (UID: \"a35d36de-bc9b-4740-b19e-9efb37df2d2f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.653892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.653864    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a35d36de-bc9b-4740-b19e-9efb37df2d2f-config-file\") pod \"limitador-limitador-7c9b87d574-h2vml\" (UID: \"a35d36de-bc9b-4740-b19e-9efb37df2d2f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.653977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.653923    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22p4l\" (UniqueName: \"kubernetes.io/projected/a35d36de-bc9b-4740-b19e-9efb37df2d2f-kube-api-access-22p4l\") pod \"limitador-limitador-7c9b87d574-h2vml\" (UID: \"a35d36de-bc9b-4740-b19e-9efb37df2d2f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.654472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.654453    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a35d36de-bc9b-4740-b19e-9efb37df2d2f-config-file\") pod \"limitador-limitador-7c9b87d574-h2vml\" (UID: \"a35d36de-bc9b-4740-b19e-9efb37df2d2f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.667599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.667570    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22p4l\" (UniqueName: \"kubernetes.io/projected/a35d36de-bc9b-4740-b19e-9efb37df2d2f-kube-api-access-22p4l\") pod \"limitador-limitador-7c9b87d574-h2vml\" (UID: \"a35d36de-bc9b-4740-b19e-9efb37df2d2f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:42.776639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:42.776587    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:43.047886 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.047813    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:12:43.050668 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:12:43.050641    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35d36de_bc9b_4740_b19e_9efb37df2d2f.slice/crio-d68c78b5645fb5b2796612c64d34808d662900ea9f22374e434e08de043856dd WatchSource:0}: Error finding container d68c78b5645fb5b2796612c64d34808d662900ea9f22374e434e08de043856dd: Status 404 returned error can't find the container with id d68c78b5645fb5b2796612c64d34808d662900ea9f22374e434e08de043856dd
Oct 30 20:12:43.070924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.070899    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xwm8h"]
Oct 30 20:12:43.075380 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.075364    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:12:43.083499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.083477    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x29mr\""
Oct 30 20:12:43.083775 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.083755    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xwm8h"]
Oct 30 20:12:43.158600 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.158565    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflts\" (UniqueName: \"kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts\") pod \"authorino-56c6c9ddd5-xwm8h\" (UID: \"51fb668d-477d-42a4-a725-f91adec6a438\") " pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:12:43.259362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.259337    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflts\" (UniqueName: \"kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts\") pod \"authorino-56c6c9ddd5-xwm8h\" (UID: \"51fb668d-477d-42a4-a725-f91adec6a438\") " pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:12:43.273114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.273093    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflts\" (UniqueName: \"kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts\") pod \"authorino-56c6c9ddd5-xwm8h\" (UID: \"51fb668d-477d-42a4-a725-f91adec6a438\") " pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:12:43.387744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.387716    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:12:43.658373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:43.658290    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xwm8h"]
Oct 30 20:12:43.661251 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:12:43.661223    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51fb668d_477d_42a4_a725_f91adec6a438.slice/crio-ce7424207920851e49caf7354538d07c71aae413f449beb24f3ec7b59a278ccd WatchSource:0}: Error finding container ce7424207920851e49caf7354538d07c71aae413f449beb24f3ec7b59a278ccd: Status 404 returned error can't find the container with id ce7424207920851e49caf7354538d07c71aae413f449beb24f3ec7b59a278ccd
Oct 30 20:12:44.001120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:44.001033    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h" event={"ID":"51fb668d-477d-42a4-a725-f91adec6a438","Type":"ContainerStarted","Data":"ce7424207920851e49caf7354538d07c71aae413f449beb24f3ec7b59a278ccd"}
Oct 30 20:12:44.002182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:44.002156    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml" event={"ID":"a35d36de-bc9b-4740-b19e-9efb37df2d2f","Type":"ContainerStarted","Data":"d68c78b5645fb5b2796612c64d34808d662900ea9f22374e434e08de043856dd"}
Oct 30 20:12:45.008442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:45.008393    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml" event={"ID":"a35d36de-bc9b-4740-b19e-9efb37df2d2f","Type":"ContainerStarted","Data":"fb8da35d5bafd55acd5a1383bd473a700ad601383007090213bccdaf861b3755"}
Oct 30 20:12:45.008912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:45.008529    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:12:45.028879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:45.028827    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml" podStartSLOduration=1.517793634 podStartE2EDuration="3.028814392s" totalImagesPullingTime="1.511020758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:45.027813105 +0000 UTC m=+497.931233129" watchObservedRunningTime="2026-10-30 20:12:45.028814392 +0000 UTC m=+497.932234415"
Oct 30 20:12:46.014052 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:46.013991    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h" event={"ID":"51fb668d-477d-42a4-a725-f91adec6a438","Type":"ContainerStarted","Data":"e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a"}
Oct 30 20:12:46.033588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:46.033541    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h" podStartSLOduration=1.162289393 podStartE2EDuration="3.033527776s" totalImagesPullingTime="1.871238383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:46.032973683 +0000 UTC m=+498.936393709" watchObservedRunningTime="2026-10-30 20:12:46.033527776 +0000 UTC m=+498.936947802"
Oct 30 20:12:49.653501 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:12:49.653466    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:12:56.015566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:12:56.015536    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2vml"
Oct 30 20:13:17.254136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:17.254107    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xwm8h"]
Oct 30 20:13:17.254678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:17.254271    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h" podUID="51fb668d-477d-42a4-a725-f91adec6a438" containerName="authorino" containerID="cri-o://e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a" gracePeriod=30
Oct 30 20:13:17.505474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:17.505414    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:13:17.603554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:17.603522    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts\" (UniqueName: \"kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts\") pod \"51fb668d-477d-42a4-a725-f91adec6a438\" (UID: \"51fb668d-477d-42a4-a725-f91adec6a438\") "
Oct 30 20:13:17.605516 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:17.605494    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts" pod "51fb668d-477d-42a4-a725-f91adec6a438" (UID: "51fb668d-477d-42a4-a725-f91adec6a438"). InnerVolumeSpecName "kube-api-access-gflts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:17.704526 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:17.704501    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gflts\" (UniqueName: \"kubernetes.io/projected/51fb668d-477d-42a4-a725-f91adec6a438-kube-api-access-gflts\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:18.144099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.144065    6058 generic.go:356] "Generic (PLEG): container finished" podID="51fb668d-477d-42a4-a725-f91adec6a438" containerID="e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a" exitCode=0
Oct 30 20:13:18.144249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.144114    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h"
Oct 30 20:13:18.144249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.144129    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h" event={"ID":"51fb668d-477d-42a4-a725-f91adec6a438","Type":"ContainerDied","Data":"e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a"}
Oct 30 20:13:18.144249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.144167    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xwm8h" event={"ID":"51fb668d-477d-42a4-a725-f91adec6a438","Type":"ContainerDied","Data":"ce7424207920851e49caf7354538d07c71aae413f449beb24f3ec7b59a278ccd"}
Oct 30 20:13:18.144249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.144189    6058 scope.go:122] "RemoveContainer" containerID="e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a"
Oct 30 20:13:18.153222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.153199    6058 scope.go:122] "RemoveContainer" containerID="e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a"
Oct 30 20:13:18.153473 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:13:18.153454    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a\": container with ID starting with e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a not found: ID does not exist" containerID="e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a"
Oct 30 20:13:18.153521 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.153481    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a"} err="failed to get container status \"e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a\": rpc error: code = NotFound desc = could not find container \"e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a\": container with ID starting with e2dae5cd549a51b645c2ebc9fd314a27004c3d7a30bccf05c56a3d9e8cf8a57a not found: ID does not exist"
Oct 30 20:13:18.165861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.165840    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xwm8h"]
Oct 30 20:13:18.169933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:18.169916    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xwm8h"]
Oct 30 20:13:19.210888 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:13:19.210853    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fb668d-477d-42a4-a725-f91adec6a438" path="/var/lib/kubelet/pods/51fb668d-477d-42a4-a725-f91adec6a438/volumes"
Oct 30 20:13:19.653601 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:13:19.653566    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:13:49.654758 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:13:49.654718    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:14:03.105488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:14:03.105455    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:14:19.654243 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:14:19.654205    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:14:49.654443 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:14:49.654407    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:15:00.159171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.159138    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"]
Oct 30 20:15:00.159951 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.159639    6058 cpu_manager.go:414] "containerMap: removing container" podUID="51fb668d-477d-42a4-a725-f91adec6a438" containerName="authorino"
Oct 30 20:15:00.159951 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.159660    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fb668d-477d-42a4-a725-f91adec6a438" containerName="authorino"
Oct 30 20:15:00.159951 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.159765    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fb668d-477d-42a4-a725-f91adec6a438" containerName="authorino"
Oct 30 20:15:00.162765 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.162743    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.167239 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.167216    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:15:00.168093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.168073    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:15:00.171808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.171777    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"]
Oct 30 20:15:00.209793 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.209770    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:15:00.269182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.269161    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-d2mkx\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.269284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.269204    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ldd\" (UniqueName: \"kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd\") pod \"maas-api-key-cleanup-29889855-d2mkx\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.370176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.370147    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-d2mkx\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.370274 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.370191    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ldd\" (UniqueName: \"kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd\") pod \"maas-api-key-cleanup-29889855-d2mkx\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.370733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.370714    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-d2mkx\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.385150 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.385125    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ldd\" (UniqueName: \"kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd\") pod \"maas-api-key-cleanup-29889855-d2mkx\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.475675 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.475625    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:00.745265 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.745199    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"]
Oct 30 20:15:00.748165 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:00.748129    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f025b00_6688_44aa_aa31_5ca1ac61fe91.slice/crio-7ea26d10da85aced0958956ef33d02ede889806ecdeac2a0f3898b7bebc7f13f WatchSource:0}: Error finding container 7ea26d10da85aced0958956ef33d02ede889806ecdeac2a0f3898b7bebc7f13f: Status 404 returned error can't find the container with id 7ea26d10da85aced0958956ef33d02ede889806ecdeac2a0f3898b7bebc7f13f
Oct 30 20:15:00.749975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:00.749959    6058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:15:01.574589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:01.574555    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerStarted","Data":"7ea26d10da85aced0958956ef33d02ede889806ecdeac2a0f3898b7bebc7f13f"}
Oct 30 20:15:02.262246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:02.262213    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:15:03.306954 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.306923    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"]
Oct 30 20:15:03.310602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.310578    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.319882 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.319685    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 20:15:03.319882 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.319746    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 20:15:03.320850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.320807    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 20:15:03.320976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.320912    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9h5g9\""
Oct 30 20:15:03.324784 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.324763    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"]
Oct 30 20:15:03.397336 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.397303    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.397493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.397353    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57djf\" (UniqueName: \"kubernetes.io/projected/75e73891-2ed1-4066-83f7-4f9a9d384f66-kube-api-access-57djf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.397547 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.397489    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.397617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.397551    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.397653 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.397633    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e73891-2ed1-4066-83f7-4f9a9d384f66-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.397697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.397682    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.498735 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.498694    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e73891-2ed1-4066-83f7-4f9a9d384f66-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.498920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.498761    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.498920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.498801    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.498920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.498845    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57djf\" (UniqueName: \"kubernetes.io/projected/75e73891-2ed1-4066-83f7-4f9a9d384f66-kube-api-access-57djf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.498920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.498900    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.498920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.498923    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.499232 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.499198    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.499481 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.499460    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.499687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.499666    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.501498 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.501478    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e73891-2ed1-4066-83f7-4f9a9d384f66-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.501584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.501547    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e73891-2ed1-4066-83f7-4f9a9d384f66-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.514071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.514050    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57djf\" (UniqueName: \"kubernetes.io/projected/75e73891-2ed1-4066-83f7-4f9a9d384f66-kube-api-access-57djf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5qft9\" (UID: \"75e73891-2ed1-4066-83f7-4f9a9d384f66\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.627636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.627592    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:03.920405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:03.920328    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"]
Oct 30 20:15:03.924076 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:03.924045    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e73891_2ed1_4066_83f7_4f9a9d384f66.slice/crio-e59e58cb1566743564d0a9fae540c754eece50eb223fcd7e27f6801d8a62b8e8 WatchSource:0}: Error finding container e59e58cb1566743564d0a9fae540c754eece50eb223fcd7e27f6801d8a62b8e8: Status 404 returned error can't find the container with id e59e58cb1566743564d0a9fae540c754eece50eb223fcd7e27f6801d8a62b8e8
Oct 30 20:15:04.589627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:04.589087    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerStarted","Data":"c0b04a936d565899d0c8b2fe4eae0575151c168aae425cd85aa8e8b5d3f90e0c"}
Oct 30 20:15:04.590843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:04.590817    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9" event={"ID":"75e73891-2ed1-4066-83f7-4f9a9d384f66","Type":"ContainerStarted","Data":"e59e58cb1566743564d0a9fae540c754eece50eb223fcd7e27f6801d8a62b8e8"}
Oct 30 20:15:05.282199 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.282131    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" podStartSLOduration=1.944565553 podStartE2EDuration="5.28211038s" totalImagesPullingTime="3.337544827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.608937795 +0000 UTC m=+637.512357822" watchObservedRunningTime="2026-10-30 20:15:05.28211038 +0000 UTC m=+638.185530407"
Oct 30 20:15:05.282701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.282679    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:15:05.596441 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.596400    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9" event={"ID":"75e73891-2ed1-4066-83f7-4f9a9d384f66","Type":"ContainerStarted","Data":"44e1ecdf3ef467bc041cc17b99206b63d0770e4855ef5f300474269bf1987aaa"}
Oct 30 20:15:05.598019 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.597968    6058 generic.go:356] "Generic (PLEG): container finished" podID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerID="c0b04a936d565899d0c8b2fe4eae0575151c168aae425cd85aa8e8b5d3f90e0c" exitCode=60
Oct 30 20:15:05.598143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.598029    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerDied","Data":"c0b04a936d565899d0c8b2fe4eae0575151c168aae425cd85aa8e8b5d3f90e0c"}
Oct 30 20:15:05.598230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.598192    6058 scope.go:122] "RemoveContainer" containerID="c0b04a936d565899d0c8b2fe4eae0575151c168aae425cd85aa8e8b5d3f90e0c"
Oct 30 20:15:05.620711 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.620646    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9" podStartSLOduration=1.452574584 podStartE2EDuration="2.620632495s" totalImagesPullingTime="1.168057911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:05.619409063 +0000 UTC m=+638.522829098" watchObservedRunningTime="2026-10-30 20:15:05.620632495 +0000 UTC m=+638.524052521"
Oct 30 20:15:05.628039 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:05.627994    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:06.520235 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.520207    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"]
Oct 30 20:15:06.523507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.523486    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.531137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.531114    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 20:15:06.534928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.534874    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"]
Oct 30 20:15:06.603770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.603736    6058 generic.go:356] "Generic (PLEG): container finished" podID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerID="d85c6dca6ef869d20a135c09ec38e97cbfe91a5b830f11c1765e57fc5089a315" exitCode=60
Oct 30 20:15:06.604183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.603817    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerDied","Data":"d85c6dca6ef869d20a135c09ec38e97cbfe91a5b830f11c1765e57fc5089a315"}
Oct 30 20:15:06.604183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.603867    6058 scope.go:122] "RemoveContainer" containerID="c0b04a936d565899d0c8b2fe4eae0575151c168aae425cd85aa8e8b5d3f90e0c"
Oct 30 20:15:06.604183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.604123    6058 scope.go:122] "RemoveContainer" containerID="d85c6dca6ef869d20a135c09ec38e97cbfe91a5b830f11c1765e57fc5089a315"
Oct 30 20:15:06.604434 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:15:06.604362    6058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-d2mkx_odh-ai-gateway-infra(6f025b00-6688-44aa-aa31-5ca1ac61fe91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91"
Oct 30 20:15:06.627931 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.627900    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.628038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.627941    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.628082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.628037    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.628082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.628061    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.628082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.628079    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19f9af-b5c9-4122-a0e4-20f215eef257-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.628189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.628176    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgh4\" (UniqueName: \"kubernetes.io/projected/6a19f9af-b5c9-4122-a0e4-20f215eef257-kube-api-access-zwgh4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.728870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.728836    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.728939    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.728966    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729267 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.729244    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19f9af-b5c9-4122-a0e4-20f215eef257-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.729357    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.729365    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgh4\" (UniqueName: \"kubernetes.io/projected/6a19f9af-b5c9-4122-a0e4-20f215eef257-kube-api-access-zwgh4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.729446    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729753 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.729730    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.729977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.729948    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.731215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.731196    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19f9af-b5c9-4122-a0e4-20f215eef257-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.731827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.731808    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19f9af-b5c9-4122-a0e4-20f215eef257-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.748362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.748337    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgh4\" (UniqueName: \"kubernetes.io/projected/6a19f9af-b5c9-4122-a0e4-20f215eef257-kube-api-access-zwgh4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m8gtk\" (UID: \"6a19f9af-b5c9-4122-a0e4-20f215eef257\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:06.838683 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:06.838644    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:07.116221 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.116187    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"]
Oct 30 20:15:07.119386 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:07.119358    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a19f9af_b5c9_4122_a0e4_20f215eef257.slice/crio-0f599b727a784025ef8d46db72e81ec00a35032ecf8c5d6ef491821b9ed19855 WatchSource:0}: Error finding container 0f599b727a784025ef8d46db72e81ec00a35032ecf8c5d6ef491821b9ed19855: Status 404 returned error can't find the container with id 0f599b727a784025ef8d46db72e81ec00a35032ecf8c5d6ef491821b9ed19855
Oct 30 20:15:07.326620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.326540    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:15:07.609403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.609369    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk" event={"ID":"6a19f9af-b5c9-4122-a0e4-20f215eef257","Type":"ContainerStarted","Data":"b5acd3c70d1839af99d861b82b1a6b54abc034fa3099d3e0c5c0c357af2e1888"}
Oct 30 20:15:07.609813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.609427    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk" event={"ID":"6a19f9af-b5c9-4122-a0e4-20f215eef257","Type":"ContainerStarted","Data":"0f599b727a784025ef8d46db72e81ec00a35032ecf8c5d6ef491821b9ed19855"}
Oct 30 20:15:07.611152 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.611129    6058 scope.go:122] "RemoveContainer" containerID="d85c6dca6ef869d20a135c09ec38e97cbfe91a5b830f11c1765e57fc5089a315"
Oct 30 20:15:07.611369 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:15:07.611350    6058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-d2mkx_odh-ai-gateway-infra(6f025b00-6688-44aa-aa31-5ca1ac61fe91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91"
Oct 30 20:15:07.630403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.630359    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk" podStartSLOduration=1.337942592 podStartE2EDuration="1.630348514s" totalImagesPullingTime="292.405922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:07.629636297 +0000 UTC m=+640.533056324" watchObservedRunningTime="2026-10-30 20:15:07.630348514 +0000 UTC m=+640.533768562"
Oct 30 20:15:07.838956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:07.838921    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:09.333538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.333498    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:15:09.465786 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.465747    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"]
Oct 30 20:15:09.470332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.470305    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.478364 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.478332    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 20:15:09.483175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.483150    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"]
Oct 30 20:15:09.554912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.554877    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72pf\" (UniqueName: \"kubernetes.io/projected/1a27023a-06b5-4452-bfa2-294228fd0143-kube-api-access-s72pf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.555117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.554919    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.555117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.555041    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.555117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.555095    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a27023a-06b5-4452-bfa2-294228fd0143-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.555279 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.555211    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.555279 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.555267    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.656761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.656732    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.656761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.656765    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.656875    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a27023a-06b5-4452-bfa2-294228fd0143-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.656898    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.656918    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.656945    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72pf\" (UniqueName: \"kubernetes.io/projected/1a27023a-06b5-4452-bfa2-294228fd0143-kube-api-access-s72pf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.657218    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.657240    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.657365 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.657323    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.659288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.659267    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a27023a-06b5-4452-bfa2-294228fd0143-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.659469 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.659442    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a27023a-06b5-4452-bfa2-294228fd0143-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.669991 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.669966    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72pf\" (UniqueName: \"kubernetes.io/projected/1a27023a-06b5-4452-bfa2-294228fd0143-kube-api-access-s72pf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j\" (UID: \"1a27023a-06b5-4452-bfa2-294228fd0143\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:09.784187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:09.784151    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:10.060550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.060477    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"]
Oct 30 20:15:10.063483 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:10.063453    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a27023a_06b5_4452_bfa2_294228fd0143.slice/crio-5ff40213feb6603fbf506cb0cb66c7f3c7fbe9620f737a0585c6b66b439e3cd8 WatchSource:0}: Error finding container 5ff40213feb6603fbf506cb0cb66c7f3c7fbe9620f737a0585c6b66b439e3cd8: Status 404 returned error can't find the container with id 5ff40213feb6603fbf506cb0cb66c7f3c7fbe9620f737a0585c6b66b439e3cd8
Oct 30 20:15:10.504465 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.504434    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"]
Oct 30 20:15:10.508126 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.508101    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.516491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.516474    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 20:15:10.520151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.520129    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"]
Oct 30 20:15:10.624843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.624809    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j" event={"ID":"1a27023a-06b5-4452-bfa2-294228fd0143","Type":"ContainerStarted","Data":"feaed7a8ba65142ef0a9b100f0e349832f1352f36f38919efc0c826b250d0a60"}
Oct 30 20:15:10.625038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.624861    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j" event={"ID":"1a27023a-06b5-4452-bfa2-294228fd0143","Type":"ContainerStarted","Data":"5ff40213feb6603fbf506cb0cb66c7f3c7fbe9620f737a0585c6b66b439e3cd8"}
Oct 30 20:15:10.649508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.649464    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j" podStartSLOduration=1.360000816 podStartE2EDuration="1.649451756s" totalImagesPullingTime="289.45094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:10.648379445 +0000 UTC m=+643.551799471" watchObservedRunningTime="2026-10-30 20:15:10.649451756 +0000 UTC m=+643.552871773"
Oct 30 20:15:10.667838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.667811    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2rc\" (UniqueName: \"kubernetes.io/projected/acb420cc-2386-4cca-b3ae-a53a9fb370e6-kube-api-access-gx2rc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.667971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.667850    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.667971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.667868    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.668108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.668064    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb420cc-2386-4cca-b3ae-a53a9fb370e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.668165 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.668128    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.668224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.668160    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.769713 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.769670    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb420cc-2386-4cca-b3ae-a53a9fb370e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.769713 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.769716    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.769938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.769745    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.769938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.769814    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2rc\" (UniqueName: \"kubernetes.io/projected/acb420cc-2386-4cca-b3ae-a53a9fb370e6-kube-api-access-gx2rc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.769938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.769909    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.770130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.769946    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.770206 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.770180    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.770401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.770363    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.770526 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.770419    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.772278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.772252    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb420cc-2386-4cca-b3ae-a53a9fb370e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.772408 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.772365    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb420cc-2386-4cca-b3ae-a53a9fb370e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.782816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.782786    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2rc\" (UniqueName: \"kubernetes.io/projected/acb420cc-2386-4cca-b3ae-a53a9fb370e6-kube-api-access-gx2rc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n\" (UID: \"acb420cc-2386-4cca-b3ae-a53a9fb370e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:10.784457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.784429    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:10.819814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:10.819793    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:11.097826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:11.097747    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"]
Oct 30 20:15:11.101865 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:11.101842    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacb420cc_2386_4cca_b3ae_a53a9fb370e6.slice/crio-58f247468f34076ed1d9d78b43e949568613b1aa1017f899d981ad5b3da98dc7 WatchSource:0}: Error finding container 58f247468f34076ed1d9d78b43e949568613b1aa1017f899d981ad5b3da98dc7: Status 404 returned error can't find the container with id 58f247468f34076ed1d9d78b43e949568613b1aa1017f899d981ad5b3da98dc7
Oct 30 20:15:11.630646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:11.630608    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" event={"ID":"acb420cc-2386-4cca-b3ae-a53a9fb370e6","Type":"ContainerStarted","Data":"cf80c1d526ff1dbc522968385b21b869c6bb9ac8eb1c643c3b47cc6e15e0b961"}
Oct 30 20:15:11.630646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:11.630654    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" event={"ID":"acb420cc-2386-4cca-b3ae-a53a9fb370e6","Type":"ContainerStarted","Data":"58f247468f34076ed1d9d78b43e949568613b1aa1017f899d981ad5b3da98dc7"}
Oct 30 20:15:11.651838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:11.651780    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podStartSLOduration=1.346713074 podStartE2EDuration="1.651761943s" totalImagesPullingTime="305.048869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:11.651191123 +0000 UTC m=+644.554611150" watchObservedRunningTime="2026-10-30 20:15:11.651761943 +0000 UTC m=+644.555181972"
Oct 30 20:15:11.820138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:11.820061    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:12.474722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:12.474684    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:15:13.628697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:13.628661    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:13.641902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:13.641873    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:13.652993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:13.652969    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9"
Oct 30 20:15:16.839826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:16.839788    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:16.852193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:16.852164    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:17.673438 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:17.673411    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk"
Oct 30 20:15:19.654484 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:15:19.654445    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:15:19.784503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:19.784475    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:19.796839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:19.796816    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:20.205585 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:20.205554    6058 scope.go:122] "RemoveContainer" containerID="d85c6dca6ef869d20a135c09ec38e97cbfe91a5b830f11c1765e57fc5089a315"
Oct 30 20:15:20.675569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:20.675539    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerStarted","Data":"658c44b0990dece63ff6cb4a1f9324d490abbccbb8679b86899009157075787c"}
Oct 30 20:15:20.686679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:20.686656    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j"
Oct 30 20:15:20.820423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:20.820394    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:20.832838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:20.832810    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:21.682825 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:21.682793    6058 generic.go:356] "Generic (PLEG): container finished" podID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerID="658c44b0990dece63ff6cb4a1f9324d490abbccbb8679b86899009157075787c" exitCode=60
Oct 30 20:15:21.683284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:21.682866    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerDied","Data":"658c44b0990dece63ff6cb4a1f9324d490abbccbb8679b86899009157075787c"}
Oct 30 20:15:21.683284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:21.682936    6058 scope.go:122] "RemoveContainer" containerID="d85c6dca6ef869d20a135c09ec38e97cbfe91a5b830f11c1765e57fc5089a315"
Oct 30 20:15:21.683423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:21.683407    6058 scope.go:122] "RemoveContainer" containerID="658c44b0990dece63ff6cb4a1f9324d490abbccbb8679b86899009157075787c"
Oct 30 20:15:21.684033 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:15:21.683580    6058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-d2mkx_odh-ai-gateway-infra(6f025b00-6688-44aa-aa31-5ca1ac61fe91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91"
Oct 30 20:15:21.696612 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:21.696592    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:15:21.715108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:21.715078    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"]
Oct 30 20:15:22.817914 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.817886    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:22.870397 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.870368    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd\" (UniqueName: \"kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd\") pod \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") "
Oct 30 20:15:22.870525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.870510    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca\") pod \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\" (UID: \"6f025b00-6688-44aa-aa31-5ca1ac61fe91\") "
Oct 30 20:15:22.870885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.870861    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca" pod "6f025b00-6688-44aa-aa31-5ca1ac61fe91" (UID: "6f025b00-6688-44aa-aa31-5ca1ac61fe91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:22.872504 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.872479    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd" pod "6f025b00-6688-44aa-aa31-5ca1ac61fe91" (UID: "6f025b00-6688-44aa-aa31-5ca1ac61fe91"). InnerVolumeSpecName "kube-api-access-s6ldd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:22.971881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.971802    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ldd\" (UniqueName: \"kubernetes.io/projected/6f025b00-6688-44aa-aa31-5ca1ac61fe91-kube-api-access-s6ldd\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.971881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:22.971826    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f025b00-6688-44aa-aa31-5ca1ac61fe91-maas-api-service-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.624837 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.624805    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f7d6c68cd-mbtq4"]
Oct 30 20:15:23.625163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625149    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625164    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625175    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625180    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625198    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625203    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625260    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625270    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.625386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.625276    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" containerName="cleanup"
Oct 30 20:15:23.628327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.628310    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.638858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.638828    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7d6c68cd-mbtq4"]
Oct 30 20:15:23.639052 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.639033    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x29mr\""
Oct 30 20:15:23.639115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.639092    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 20:15:23.677386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.677357    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b85d4a-009b-42d2-96a8-08391753c3d1-tls-cert\") pod \"authorino-7f7d6c68cd-mbtq4\" (UID: \"d9b85d4a-009b-42d2-96a8-08391753c3d1\") " pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.677538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.677395    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrq6\" (UniqueName: \"kubernetes.io/projected/d9b85d4a-009b-42d2-96a8-08391753c3d1-kube-api-access-wxrq6\") pod \"authorino-7f7d6c68cd-mbtq4\" (UID: \"d9b85d4a-009b-42d2-96a8-08391753c3d1\") " pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.697071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.697045    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx" event={"ID":"6f025b00-6688-44aa-aa31-5ca1ac61fe91","Type":"ContainerDied","Data":"7ea26d10da85aced0958956ef33d02ede889806ecdeac2a0f3898b7bebc7f13f"}
Oct 30 20:15:23.697193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.697072    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"
Oct 30 20:15:23.697193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.697109    6058 scope.go:122] "RemoveContainer" containerID="658c44b0990dece63ff6cb4a1f9324d490abbccbb8679b86899009157075787c"
Oct 30 20:15:23.716742 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.716722    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"]
Oct 30 20:15:23.723538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.723509    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-d2mkx"]
Oct 30 20:15:23.778529 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.778504    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b85d4a-009b-42d2-96a8-08391753c3d1-tls-cert\") pod \"authorino-7f7d6c68cd-mbtq4\" (UID: \"d9b85d4a-009b-42d2-96a8-08391753c3d1\") " pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.778602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.778540    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrq6\" (UniqueName: \"kubernetes.io/projected/d9b85d4a-009b-42d2-96a8-08391753c3d1-kube-api-access-wxrq6\") pod \"authorino-7f7d6c68cd-mbtq4\" (UID: \"d9b85d4a-009b-42d2-96a8-08391753c3d1\") " pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.780834 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.780818    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b85d4a-009b-42d2-96a8-08391753c3d1-tls-cert\") pod \"authorino-7f7d6c68cd-mbtq4\" (UID: \"d9b85d4a-009b-42d2-96a8-08391753c3d1\") " pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.791919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.791893    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrq6\" (UniqueName: \"kubernetes.io/projected/d9b85d4a-009b-42d2-96a8-08391753c3d1-kube-api-access-wxrq6\") pod \"authorino-7f7d6c68cd-mbtq4\" (UID: \"d9b85d4a-009b-42d2-96a8-08391753c3d1\") " pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:23.940203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:23.940134    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4"
Oct 30 20:15:24.218058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:24.217892    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7d6c68cd-mbtq4"]
Oct 30 20:15:24.220809 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:24.220782    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9b85d4a_009b_42d2_96a8_08391753c3d1.slice/crio-0194d331e59366a66659e83df21d6b9169afeeb6acd7d3e48835acaf20aa3dcb WatchSource:0}: Error finding container 0194d331e59366a66659e83df21d6b9169afeeb6acd7d3e48835acaf20aa3dcb: Status 404 returned error can't find the container with id 0194d331e59366a66659e83df21d6b9169afeeb6acd7d3e48835acaf20aa3dcb
Oct 30 20:15:24.702691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:24.702662    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4" event={"ID":"d9b85d4a-009b-42d2-96a8-08391753c3d1","Type":"ContainerStarted","Data":"0194d331e59366a66659e83df21d6b9169afeeb6acd7d3e48835acaf20aa3dcb"}
Oct 30 20:15:25.211149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:25.211121    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f025b00-6688-44aa-aa31-5ca1ac61fe91" path="/var/lib/kubelet/pods/6f025b00-6688-44aa-aa31-5ca1ac61fe91/volumes"
Oct 30 20:15:25.709252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:25.709206    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4" event={"ID":"d9b85d4a-009b-42d2-96a8-08391753c3d1","Type":"ContainerStarted","Data":"feddd09305649949b12afaef252bbd2681d5e827e710499454beaeefca9b9bd9"}
Oct 30 20:15:25.734398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:25.734346    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7f7d6c68cd-mbtq4" podStartSLOduration=1.772502689 podStartE2EDuration="2.734333086s" totalImagesPullingTime="961.830397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:25.73353941 +0000 UTC m=+658.636959436" watchObservedRunningTime="2026-10-30 20:15:25.734333086 +0000 UTC m=+658.637753112"
Oct 30 20:15:47.113854 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.113808    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"]
Oct 30 20:15:47.117520 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.117497    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.122907 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.122881    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9k4qv\""
Oct 30 20:15:47.125534 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.125514    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:15:47.132435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.132416    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"]
Oct 30 20:15:47.174646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174621    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.174785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174655    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.174785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174681    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.174870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174818    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.174870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174848    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.174943 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174903    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.174984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.174965    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.175048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.175019    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbql\" (UniqueName: \"kubernetes.io/projected/dc5e977f-8820-4bd8-a733-4c35785d3fc5-kube-api-access-vwbql\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.175112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.175087    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276575    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276663    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276695    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276718    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276829 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276800    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276839    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276875    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.276973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276925    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.277055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.276974    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbql\" (UniqueName: \"kubernetes.io/projected/dc5e977f-8820-4bd8-a733-4c35785d3fc5-kube-api-access-vwbql\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.277117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.277098    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.277294 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.277270    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.277393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.277375    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.277487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.277467    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.277613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.277597    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.279274 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.279245    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.279477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.279459    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.293038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.292999    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5e977f-8820-4bd8-a733-4c35785d3fc5-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.293346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.293326    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbql\" (UniqueName: \"kubernetes.io/projected/dc5e977f-8820-4bd8-a733-4c35785d3fc5-kube-api-access-vwbql\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ls9cm\" (UID: \"dc5e977f-8820-4bd8-a733-4c35785d3fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.431446 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.431371    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:47.714461 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.714393    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"]
Oct 30 20:15:47.716495 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:15:47.716461    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5e977f_8820_4bd8_a733_4c35785d3fc5.slice/crio-4cd632698176871afe9a3507abca94238619d9538e1f3f5295b8167865f26014 WatchSource:0}: Error finding container 4cd632698176871afe9a3507abca94238619d9538e1f3f5295b8167865f26014: Status 404 returned error can't find the container with id 4cd632698176871afe9a3507abca94238619d9538e1f3f5295b8167865f26014
Oct 30 20:15:47.798744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:47.798712    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" event={"ID":"dc5e977f-8820-4bd8-a733-4c35785d3fc5","Type":"ContainerStarted","Data":"4cd632698176871afe9a3507abca94238619d9538e1f3f5295b8167865f26014"}
Oct 30 20:15:49.658672 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:15:49.658637    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:15:50.765222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:50.765193    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:15:50.765484 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:50.765260    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:15:50.765484 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:50.765287    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:15:51.820142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:51.820104    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" event={"ID":"dc5e977f-8820-4bd8-a733-4c35785d3fc5","Type":"ContainerStarted","Data":"fbf1c9613805358a6ffa6ed33a6f5248e697d91b472ba039af528f804a8be694"}
Oct 30 20:15:51.845893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:51.845843    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podStartSLOduration=1.800199165 podStartE2EDuration="4.845830402s" totalImagesPullingTime="3.045631237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:51.844808422 +0000 UTC m=+684.748228449" watchObservedRunningTime="2026-10-30 20:15:51.845830402 +0000 UTC m=+684.749250422"
Oct 30 20:15:52.432182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:52.432147    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:52.433456 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:52.433437    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 20:15:52.433523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:52.433479    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 20:15:53.432351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:53.432316    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 20:15:53.432733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:53.432381    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 20:15:54.431757 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:54.431717    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 20:15:54.431923 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:54.431781    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 20:15:55.432187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:55.432151    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 20:15:55.432572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:55.432220    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 20:15:56.431735 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:56.431699    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 20:15:56.431919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:56.431759    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 20:15:57.432424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:57.432383    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 20:15:57.432815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:57.432463    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 20:15:58.911220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:58.911183    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:58.911976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:58.911951    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:15:58.912728 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:15:58.912706    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm"
Oct 30 20:16:02.156771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.156729    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"]
Oct 30 20:16:02.160700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.160675    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.175937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.175909    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"]
Oct 30 20:16:02.194967 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.194936    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"]
Oct 30 20:16:02.198911 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.198889    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.219327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.219308    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"]
Oct 30 20:16:02.328702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328677    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.328816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328706    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.328816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328725    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.328816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328796    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.328960 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328853    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.329046 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328947    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329046 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.328998    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329056    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329091    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.329263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329155    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hw2\" (UniqueName: \"kubernetes.io/projected/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-kube-api-access-p7hw2\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329187    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329216    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhm94\" (UniqueName: \"kubernetes.io/projected/cad85ff2-3b19-4098-aeb6-50feed17e4aa-kube-api-access-xhm94\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.329442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329269    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.329442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329293    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329327    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.329442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329349    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329500    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.329619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.329536    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430347    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430373    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430391    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430411    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430433    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hw2\" (UniqueName: \"kubernetes.io/projected/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-kube-api-access-p7hw2\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430461    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430491    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm94\" (UniqueName: \"kubernetes.io/projected/cad85ff2-3b19-4098-aeb6-50feed17e4aa-kube-api-access-xhm94\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430519    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430555    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430586    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430613    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430651    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430685    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430750    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430779    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430804    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430836    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430861    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.430959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.430890    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.431468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431177    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.431468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431384    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.431468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431442    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.431636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431473    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.431636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431515    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.431753 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431650    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.431753 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431664    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.431855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431758    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.431855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.431792    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.433038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.432993    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.433164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.433147    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.433485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.433462    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.433611 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.433594    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.443996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.443973    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhm94\" (UniqueName: \"kubernetes.io/projected/cad85ff2-3b19-4098-aeb6-50feed17e4aa-kube-api-access-xhm94\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.444183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.444164    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.444341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.444325    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad85ff2-3b19-4098-aeb6-50feed17e4aa-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jrd57\" (UID: \"cad85ff2-3b19-4098-aeb6-50feed17e4aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.444424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.444408    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hw2\" (UniqueName: \"kubernetes.io/projected/b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a-kube-api-access-p7hw2\") pod \"maas-default-gateway-openshift-default-64d4cc998c-ws6r4\" (UID: \"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.473826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.473798    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:02.512147 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.512113    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:02.757204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.757180    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"]
Oct 30 20:16:02.759142 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:02.759113    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51e8d7e_f49e_4c44_9f8e_ce56a09c8a2a.slice/crio-0d821af340f383098b3edb3c094519263f10ed674f662de4c9cbb76fe98253d7 WatchSource:0}: Error finding container 0d821af340f383098b3edb3c094519263f10ed674f662de4c9cbb76fe98253d7: Status 404 returned error can't find the container with id 0d821af340f383098b3edb3c094519263f10ed674f662de4c9cbb76fe98253d7
Oct 30 20:16:02.761181 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.761150    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:02.761245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.761213    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:02.761281 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.761242    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:02.804144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.804123    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"]
Oct 30 20:16:02.805264 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:02.805244    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad85ff2_3b19_4098_aeb6_50feed17e4aa.slice/crio-bbead2b897575d5ec2048b72e61a07c2f210a93ff27309ecec1e4d0dbab92d8a WatchSource:0}: Error finding container bbead2b897575d5ec2048b72e61a07c2f210a93ff27309ecec1e4d0dbab92d8a: Status 404 returned error can't find the container with id bbead2b897575d5ec2048b72e61a07c2f210a93ff27309ecec1e4d0dbab92d8a
Oct 30 20:16:02.807341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.807273    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:02.807414 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.807360    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:02.807472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.807410    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:02.870864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.870831    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" event={"ID":"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a","Type":"ContainerStarted","Data":"bc3b9373673df9778f55cd8c961a15e311c6cb125b7b22ce01ea6081ff686a1a"}
Oct 30 20:16:02.870996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.870895    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" event={"ID":"b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a","Type":"ContainerStarted","Data":"0d821af340f383098b3edb3c094519263f10ed674f662de4c9cbb76fe98253d7"}
Oct 30 20:16:02.872610 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:02.872586    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" event={"ID":"cad85ff2-3b19-4098-aeb6-50feed17e4aa","Type":"ContainerStarted","Data":"bbead2b897575d5ec2048b72e61a07c2f210a93ff27309ecec1e4d0dbab92d8a"}
Oct 30 20:16:03.478596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:03.478557    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:03.483157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:03.483122    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:03.483378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:03.483350    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:03.891028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:03.890041    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" event={"ID":"cad85ff2-3b19-4098-aeb6-50feed17e4aa","Type":"ContainerStarted","Data":"9af2287df0fdb6de21df90ba23202894d36673efe579b1844224266213667484"}
Oct 30 20:16:03.922024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:03.921959    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podStartSLOduration=1.921945328 podStartE2EDuration="1.921945328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:02.898099654 +0000 UTC m=+695.801519676" watchObservedRunningTime="2026-10-30 20:16:03.921945328 +0000 UTC m=+696.825365351"
Oct 30 20:16:03.922543 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:03.922512    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podStartSLOduration=1.922503925 podStartE2EDuration="1.922503925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:03.920448985 +0000 UTC m=+696.823869002" watchObservedRunningTime="2026-10-30 20:16:03.922503925 +0000 UTC m=+696.825923950"
Oct 30 20:16:04.482029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:04.474426    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:04.482029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:04.474490    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:04.513302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:04.513271    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:04.514760 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:04.514739    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 20:16:04.514813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:04.514781    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 20:16:05.475032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:05.474410    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:05.475032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:05.474490    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:05.512485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:05.512448    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 20:16:05.512880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:05.512499    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 20:16:06.474806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:06.474358    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:06.474806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:06.474423    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:06.513035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:06.512989    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 20:16:06.513392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:06.513054    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 20:16:07.478031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:07.477396    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:07.478031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:07.477462    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:07.512561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:07.512527    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 20:16:07.512814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:07.512782    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 20:16:08.474537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:08.474500    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:08.475194 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:08.475148    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:08.515480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:08.515440    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 20:16:08.515759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:08.515730    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 20:16:09.474791 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:09.474753    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 20:16:09.475195 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:09.474835    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 20:16:09.513591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:09.513560    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 20:16:09.513711 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:09.513625    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 20:16:11.151874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:11.151835    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:11.152373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:11.152322    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:11.153204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:11.153180    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4"
Oct 30 20:16:11.164743 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:11.164719    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:11.165109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:11.165078    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:11.165880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:11.165859    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57"
Oct 30 20:16:19.654303 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:16:19.654265    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:16:24.355947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.355910    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"]
Oct 30 20:16:24.364990 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.364963    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.374934 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.374908    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"]
Oct 30 20:16:24.375308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.375258    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c646da-openshift-default-dockercfg-2gm5m\""
Oct 30 20:16:24.476403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476370    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476423    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp664\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476616 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476552    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476616 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476600    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476709    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476777 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476759    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476799    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476859    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.476921 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.476908    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.560437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.560405    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"]
Oct 30 20:16:24.564185 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.564164    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.578132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578106    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"]
Oct 30 20:16:24.578661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578420    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578487    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578536    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578597    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578624    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578657    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578697    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp664\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578739    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.578997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.578776    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.579237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.579212    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.579572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.579552    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.579912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.579886    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.580880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.580811    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.581247 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.581226    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.582295 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.582250    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.582800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.582772    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.584788 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.584765    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-909519-openshift-default-dockercfg-r5l5q\""
Oct 30 20:16:24.596947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.596919    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp664\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.598144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.598121    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.679532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679491    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.679739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679548    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.679739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679598    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.679739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679655    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhqd\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.679739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679685    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.679739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679715    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.679739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679728    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:24.680065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679743    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.680065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679770    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.680065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.679833    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781140    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781205    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781249    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781304    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781375    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhqd\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781410    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781439    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781468    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781496    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781600    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781647    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.781904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.781882    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.782099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.782077    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.782375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.782353    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.784408 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.784386    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.784591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.784567    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.790175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.790150    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"]
Oct 30 20:16:24.797949 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.797933    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.799118 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.799075    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.800677 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.800653    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhqd\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.807343 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.807312    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"]
Oct 30 20:16:24.814069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.814045    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-75e734-openshift-default-dockercfg-dv849\""
Oct 30 20:16:24.878850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.878823    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:24.882048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882026    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882064    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882163    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggcv\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882243 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882213    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882333    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882360    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882382    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882431    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.882560 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.882483    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.978878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.977752    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"]
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984030    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984080    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984122    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984155    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984227    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984261    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984314    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggcv\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984347    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.984849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.984439    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.988207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.987827    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.988207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.987885    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.988207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.988095    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.988207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.988169    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.988381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.988212    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.989171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.989152    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.996663 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.995527    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:24.997543 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.997510    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:24.997648 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.997579    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:24.997648 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:24.997607    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:25.001319 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.001297    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" event={"ID":"2210b840-941b-4d46-8777-86aeabd3af6f","Type":"ContainerStarted","Data":"a3ee297af2ad6a5480ef1391c44db87cd8ca55a890629c796eb4755a7173eb92"}
Oct 30 20:16:25.003687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.003666    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:25.003836 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.003816    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggcv\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:25.117302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.117271    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:25.166396 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.166360    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"]
Oct 30 20:16:25.172493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.172462    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:25.172614 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.172527    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:25.172614 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.172558    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:25.415510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.415431    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"]
Oct 30 20:16:25.417978 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:25.417947    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc843d3e_f474_49d9_8903_101e48aa44ae.slice/crio-70eb71f666a0731a0360447369f3ac5d173ef6277faeb44a9f2f833ec5fe8d05 WatchSource:0}: Error finding container 70eb71f666a0731a0360447369f3ac5d173ef6277faeb44a9f2f833ec5fe8d05: Status 404 returned error can't find the container with id 70eb71f666a0731a0360447369f3ac5d173ef6277faeb44a9f2f833ec5fe8d05
Oct 30 20:16:25.420986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.420956    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:25.421125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.421102    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:25.421187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:25.421156    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:26.006880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.006842    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" event={"ID":"e4b2a700-f272-4d97-99e7-e894bf75f21f","Type":"ContainerStarted","Data":"cb913287298ce1002a3a7ae7c778c96cfaf34ddd93aac1a12f7d790ea870157f"}
Oct 30 20:16:26.007083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.006905    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" event={"ID":"e4b2a700-f272-4d97-99e7-e894bf75f21f","Type":"ContainerStarted","Data":"521bfc5d95dd5d779f93f7c10bb00278a34883a975e642bf3fba4191cc2c00ea"}
Oct 30 20:16:26.008561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.008527    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" event={"ID":"2210b840-941b-4d46-8777-86aeabd3af6f","Type":"ContainerStarted","Data":"99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"}
Oct 30 20:16:26.010137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.010111    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" event={"ID":"dc843d3e-f474-49d9-8903-101e48aa44ae","Type":"ContainerStarted","Data":"6201ef598fee2960c8116ef20bde6efc3f2efc93689858841a8a704ab62764d3"}
Oct 30 20:16:26.010221 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.010156    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" event={"ID":"dc843d3e-f474-49d9-8903-101e48aa44ae","Type":"ContainerStarted","Data":"70eb71f666a0731a0360447369f3ac5d173ef6277faeb44a9f2f833ec5fe8d05"}
Oct 30 20:16:26.043915 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.043873    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podStartSLOduration=2.043862121 podStartE2EDuration="2.043862121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:26.042672808 +0000 UTC m=+718.946092832" watchObservedRunningTime="2026-10-30 20:16:26.043862121 +0000 UTC m=+718.947282212"
Oct 30 20:16:26.076936 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.076863    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podStartSLOduration=2.076847595 podStartE2EDuration="2.076847595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:26.075830458 +0000 UTC m=+718.979250483" watchObservedRunningTime="2026-10-30 20:16:26.076847595 +0000 UTC m=+718.980267622"
Oct 30 20:16:26.101725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.101671    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podStartSLOduration=2.101658951 podStartE2EDuration="2.101658951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:26.100440328 +0000 UTC m=+719.003860367" watchObservedRunningTime="2026-10-30 20:16:26.101658951 +0000 UTC m=+719.005078976"
Oct 30 20:16:26.117667 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.117641    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:26.123114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.123090    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:26.681167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.681138    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:26.686739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.686713    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:26.881060 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.880715    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:26.886375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:26.886348    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:27.018559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:27.018461    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:27.019468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:27.018988    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:27.019468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:27.019052    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:27.019468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:27.019415    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:27.019660 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:27.019472    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:27.019660 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:27.019510    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:30.495415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.495380    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"]
Oct 30 20:16:30.499676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.499657    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.508905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.508878    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"]
Oct 30 20:16:30.571142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.571107    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-44135b\""
Oct 30 20:16:30.571344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.571246    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-44135b-dockercfg-mdst7\""
Oct 30 20:16:30.643862 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.643832    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.644053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.643923    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vbb\" (UniqueName: \"kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb\") pod \"payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.744669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.744629    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vbb\" (UniqueName: \"kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb\") pod \"payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.744811 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.744710    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.745441 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.745398    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.758573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.758539    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vbb\" (UniqueName: \"kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb\") pod \"payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:30.813225 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:30.813202    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:31.092697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.092625    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"]
Oct 30 20:16:31.095842 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:31.095806    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ba866f_d418_4fd2_b23f_de771e91a6f9.slice/crio-4eed86cef9c171ed0d99b5ed6d895c725f197dd1512777e1a4503c4af22a5e7c WatchSource:0}: Error finding container 4eed86cef9c171ed0d99b5ed6d895c725f197dd1512777e1a4503c4af22a5e7c: Status 404 returned error can't find the container with id 4eed86cef9c171ed0d99b5ed6d895c725f197dd1512777e1a4503c4af22a5e7c
Oct 30 20:16:31.822881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.822845    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"]
Oct 30 20:16:31.826631 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.826611    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:31.848846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.848819    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5197ec-dockercfg-q4tkg\""
Oct 30 20:16:31.850409 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.850386    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"]
Oct 30 20:16:31.896702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.896678    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5197ec\""
Oct 30 20:16:31.958389 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.958334    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcpc4\" (UniqueName: \"kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4\") pod \"payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:31.958389 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:31.958385    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:32.044273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.044231    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" event={"ID":"27ba866f-d418-4fd2-b23f-de771e91a6f9","Type":"ContainerStarted","Data":"4eed86cef9c171ed0d99b5ed6d895c725f197dd1512777e1a4503c4af22a5e7c"}
Oct 30 20:16:32.060029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.059967    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcpc4\" (UniqueName: \"kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4\") pod \"payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:32.060193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.060040    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:32.060901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.060865    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:32.075619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.075552    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcpc4\" (UniqueName: \"kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4\") pod \"payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:32.142118 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.142080    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:32.309242 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.309211    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"]
Oct 30 20:16:32.314163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.314138    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.325447 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.325421    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"]
Oct 30 20:16:32.326327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.326253    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-75e734\""
Oct 30 20:16:32.363785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.363748    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jkz\" (UniqueName: \"kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz\") pod \"payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.363953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.363883    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.427565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.427534    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"]
Oct 30 20:16:32.431623 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:32.431581    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb6e6b6_7442_4a01_97f2_8e967b489849.slice/crio-cffb0febf45d08e7df9e256135f932ebe7a3639c097e59edef534c726931f83d WatchSource:0}: Error finding container cffb0febf45d08e7df9e256135f932ebe7a3639c097e59edef534c726931f83d: Status 404 returned error can't find the container with id cffb0febf45d08e7df9e256135f932ebe7a3639c097e59edef534c726931f83d
Oct 30 20:16:32.466690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.465192    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jkz\" (UniqueName: \"kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz\") pod \"payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.466690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.465343    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.466690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.466636    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.481444 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.481400    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jkz\" (UniqueName: \"kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz\") pod \"payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.634724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.634519    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:32.927983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:32.927952    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"]
Oct 30 20:16:32.929201 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:32.929175    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcf388f_663e_47a2_9cd7_d38c9b158a4c.slice/crio-6ccfc84fc04aa931cb13b2d2c4f198269bd9663e6b48086c299a8af173151b85 WatchSource:0}: Error finding container 6ccfc84fc04aa931cb13b2d2c4f198269bd9663e6b48086c299a8af173151b85: Status 404 returned error can't find the container with id 6ccfc84fc04aa931cb13b2d2c4f198269bd9663e6b48086c299a8af173151b85
Oct 30 20:16:33.051773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.051356    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" event={"ID":"7fcf388f-663e-47a2-9cd7-d38c9b158a4c","Type":"ContainerStarted","Data":"6ccfc84fc04aa931cb13b2d2c4f198269bd9663e6b48086c299a8af173151b85"}
Oct 30 20:16:33.053846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.053399    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" event={"ID":"5eb6e6b6-7442-4a01-97f2-8e967b489849","Type":"ContainerStarted","Data":"cffb0febf45d08e7df9e256135f932ebe7a3639c097e59edef534c726931f83d"}
Oct 30 20:16:33.206801 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.206713    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"]
Oct 30 20:16:33.213495 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.213467    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.233741 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.230767    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"]
Oct 30 20:16:33.240401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.240367    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c646da\""
Oct 30 20:16:33.265789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.265744    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:16:33.282411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.282281    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhzp\" (UniqueName: \"kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp\") pod \"payload-processing-e2e-worker-w0-c646da-8464897474-nxf54\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.282411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.282320    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c646da-8464897474-nxf54\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.292388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.292352    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c646da-dockercfg-4w4qh\""
Oct 30 20:16:33.385595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.383842    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhzp\" (UniqueName: \"kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp\") pod \"payload-processing-e2e-worker-w0-c646da-8464897474-nxf54\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.385595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.383893    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c646da-8464897474-nxf54\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.385595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.384833    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c646da-8464897474-nxf54\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.417434 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.417387    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhzp\" (UniqueName: \"kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp\") pod \"payload-processing-e2e-worker-w0-c646da-8464897474-nxf54\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.553068 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.552491    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:33.915282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:33.915244    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"]
Oct 30 20:16:34.033394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.033359    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"]
Oct 30 20:16:34.037373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.037351    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.048972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.048949    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"]
Oct 30 20:16:34.087707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.087676    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" event={"ID":"edd00c5f-7a1e-45b2-a6d0-10471794c402","Type":"ContainerStarted","Data":"80f5e894724df5baf885cab2c53da618f3415298d4644f81a7573faedd1d2be9"}
Oct 30 20:16:34.095262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.095228    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.095588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.095566    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sld\" (UniqueName: \"kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld\") pod \"payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.107739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.107707    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-909519-dockercfg-7bzmg\""
Oct 30 20:16:34.107867 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.107802    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-909519\""
Oct 30 20:16:34.202876 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.202827    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sld\" (UniqueName: \"kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld\") pod \"payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.203023 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.202925    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.203823 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.203797    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.221744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.220212    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sld\" (UniqueName: \"kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld\") pod \"payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.365207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.365182    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:34.710934 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:34.710904    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"]
Oct 30 20:16:34.713047 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:34.712986    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfd3442_ee08_4689_a722_ec4e41e90ab4.slice/crio-5398f47f9df0d1674f612e788d22c21d8672ad0fc240845f1afd64a015e940a3 WatchSource:0}: Error finding container 5398f47f9df0d1674f612e788d22c21d8672ad0fc240845f1afd64a015e940a3: Status 404 returned error can't find the container with id 5398f47f9df0d1674f612e788d22c21d8672ad0fc240845f1afd64a015e940a3
Oct 30 20:16:35.094448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.094368    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" event={"ID":"7fcf388f-663e-47a2-9cd7-d38c9b158a4c","Type":"ContainerStarted","Data":"e4bc98ed92643d0df4bab799b255d7453b6ca550191d87ca2962bbb58f7318be"}
Oct 30 20:16:35.094883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.094474    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:35.096042 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.095998    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" event={"ID":"edd00c5f-7a1e-45b2-a6d0-10471794c402","Type":"ContainerStarted","Data":"1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6"}
Oct 30 20:16:35.096290 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.096125    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:35.097786 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.097761    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" event={"ID":"5eb6e6b6-7442-4a01-97f2-8e967b489849","Type":"ContainerStarted","Data":"072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061"}
Oct 30 20:16:35.097899 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.097806    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:35.099183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.099160    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" event={"ID":"27ba866f-d418-4fd2-b23f-de771e91a6f9","Type":"ContainerStarted","Data":"6d7500cdaba04156b05daa20ecb354fbd0cfd1c4237e73139ed3286023765231"}
Oct 30 20:16:35.099326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.099212    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:35.100752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.100731    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" event={"ID":"3bfd3442-ee08-4689-a722-ec4e41e90ab4","Type":"ContainerStarted","Data":"36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61"}
Oct 30 20:16:35.100852 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.100764    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" event={"ID":"3bfd3442-ee08-4689-a722-ec4e41e90ab4","Type":"ContainerStarted","Data":"5398f47f9df0d1674f612e788d22c21d8672ad0fc240845f1afd64a015e940a3"}
Oct 30 20:16:35.100921 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.100864    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:35.113760 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.113715    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" podStartSLOduration=1.185791698 podStartE2EDuration="3.113699153s" totalImagesPullingTime="1.927907455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:35.112656082 +0000 UTC m=+728.016076111" watchObservedRunningTime="2026-10-30 20:16:35.113699153 +0000 UTC m=+728.017119180"
Oct 30 20:16:35.158149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.158100    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" podStartSLOduration=1.158085174 podStartE2EDuration="1.158085174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:35.15742373 +0000 UTC m=+728.060843758" watchObservedRunningTime="2026-10-30 20:16:35.158085174 +0000 UTC m=+728.061505202"
Oct 30 20:16:35.159199 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.159161    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" podStartSLOduration=2.200059061 podStartE2EDuration="5.159151192s" totalImagesPullingTime="2.959092131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:35.139157472 +0000 UTC m=+728.042577500" watchObservedRunningTime="2026-10-30 20:16:35.159151192 +0000 UTC m=+728.062571217"
Oct 30 20:16:35.176038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.175967    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" podStartSLOduration=1.746314334 podStartE2EDuration="2.175951702s" totalImagesPullingTime="429.637368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:35.1756045 +0000 UTC m=+728.079024539" watchObservedRunningTime="2026-10-30 20:16:35.175951702 +0000 UTC m=+728.079371730"
Oct 30 20:16:35.194873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:35.194833    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" podStartSLOduration=1.875270384 podStartE2EDuration="4.194824777s" totalImagesPullingTime="2.319554393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:35.193565841 +0000 UTC m=+728.096985868" watchObservedRunningTime="2026-10-30 20:16:35.194824777 +0000 UTC m=+728.098244803"
Oct 30 20:16:37.464594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:37.464512    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:16:42.658030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:42.657609    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:16:46.109648 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:46.109478    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:16:46.110159 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:46.110093    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:16:46.110240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:46.110175    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:16:46.110565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:46.110500    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:16:46.110843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:46.110817    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:16:49.212888 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:49.212862    6058 generic.go:356] "Generic (PLEG): container finished" podID="2210b840-941b-4d46-8777-86aeabd3af6f" containerID="99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8" exitCode=0
Oct 30 20:16:49.213644 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:49.213615    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" event={"ID":"2210b840-941b-4d46-8777-86aeabd3af6f","Type":"ContainerDied","Data":"99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"}
Oct 30 20:16:49.214104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:49.214088    6058 scope.go:122] "RemoveContainer" containerID="99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"
Oct 30 20:16:49.215567 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:49.215540    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:49.215656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:49.215604    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:49.215656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:49.215638    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:49.684141 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:16:49.684079    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:16:50.226206 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.226161    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" event={"ID":"2210b840-941b-4d46-8777-86aeabd3af6f","Type":"ContainerStarted","Data":"7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037"}
Oct 30 20:16:50.227821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.227790    6058 generic.go:356] "Generic (PLEG): container finished" podID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerID="6201ef598fee2960c8116ef20bde6efc3f2efc93689858841a8a704ab62764d3" exitCode=0
Oct 30 20:16:50.227947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.227857    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" event={"ID":"dc843d3e-f474-49d9-8903-101e48aa44ae","Type":"ContainerDied","Data":"6201ef598fee2960c8116ef20bde6efc3f2efc93689858841a8a704ab62764d3"}
Oct 30 20:16:50.228196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.228182    6058 scope.go:122] "RemoveContainer" containerID="6201ef598fee2960c8116ef20bde6efc3f2efc93689858841a8a704ab62764d3"
Oct 30 20:16:50.229595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.229559    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:50.229712 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.229647    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:50.229773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.229715    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:50.685658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.685623    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:50.685983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.685775    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 20:16:50.686180 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:50.686152    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 20:16:51.236956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:51.236904    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" event={"ID":"dc843d3e-f474-49d9-8903-101e48aa44ae","Type":"ContainerStarted","Data":"a1065fabdd060f82d5a1d0b97f78b6174e37d9651c851880b305d61958ebebd2"}
Oct 30 20:16:51.689619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:51.689577    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 20:16:51.689817 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:51.689639    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 20:16:52.117855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:52.117803    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:52.118555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:52.118167    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:16:52.118555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:52.118244    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 20:16:52.696525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:52.696487    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 20:16:52.697120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:52.697085    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 20:16:53.118569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:53.118534    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:16:53.118838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:53.118802    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 20:16:53.692413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:53.692374    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 20:16:53.693964 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:53.693601    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 20:16:53.997082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:53.994915    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:16:54.121344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:54.121298    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:16:54.121532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:54.121380    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 20:16:54.695768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:54.695342    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 20:16:54.695768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:54.695402    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 20:16:55.119840 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.119403    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:16:55.119840 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.119482    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 20:16:55.832825 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.830829    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"]
Oct 30 20:16:55.840341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.840314    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"]
Oct 30 20:16:55.840496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.840405    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:55.846671 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.846640    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-75e734\"/\"default-dockercfg-kftsm\""
Oct 30 20:16:55.846786 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.846738    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-75e734\"/\"simulator-w6-75e734-kserve-self-signed-certs\""
Oct 30 20:16:55.846842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.846827    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-75e734\"/\"openshift-service-ca.crt\""
Oct 30 20:16:55.846944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.846929    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-75e734\"/\"kube-root-ca.crt\""
Oct 30 20:16:55.935604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.935152    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llsbv\" (UniqueName: \"kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:55.935604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.935207    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:55.935604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.935314    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:55.935604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.935358    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:55.935604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.935390    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:55.935604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:55.935426    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.033981 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.033196    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:56.033981 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.033930    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.036092    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.036157    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.036190    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.036232    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.036278    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llsbv\" (UniqueName: \"kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.036307    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.038948    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.039049    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:16:56.039568 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.039257    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.039568 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.039424    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.041084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.039997    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.042572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.042547    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.056643 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.055075    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llsbv\" (UniqueName: \"kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv\") pod \"simulator-w6-75e734-kserve-69c99c89c6-5b6dt\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") " pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.117972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.117934    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:16:56.118259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.118224    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 20:16:56.202694 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.202349    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:56.540498 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:56.540406    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"]
Oct 30 20:16:56.544883 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:16:56.544854    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf801a67_90b1_408d_a14b_86907900d29d.slice/crio-b66a735b5865907a4c6bcc31dd59532289af8c510f9b9e170d92f0275accd361 WatchSource:0}: Error finding container b66a735b5865907a4c6bcc31dd59532289af8c510f9b9e170d92f0275accd361: Status 404 returned error can't find the container with id b66a735b5865907a4c6bcc31dd59532289af8c510f9b9e170d92f0275accd361
Oct 30 20:16:57.018918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.018881    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:57.020105 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.020080    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 20:16:57.020235 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.020129    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 20:16:57.121929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.121900    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:57.347308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.347223    6058 generic.go:356] "Generic (PLEG): container finished" podID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerID="cb913287298ce1002a3a7ae7c778c96cfaf34ddd93aac1a12f7d790ea870157f" exitCode=0
Oct 30 20:16:57.347767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.347303    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" event={"ID":"e4b2a700-f272-4d97-99e7-e894bf75f21f","Type":"ContainerDied","Data":"cb913287298ce1002a3a7ae7c778c96cfaf34ddd93aac1a12f7d790ea870157f"}
Oct 30 20:16:57.347767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.347739    6058 scope.go:122] "RemoveContainer" containerID="cb913287298ce1002a3a7ae7c778c96cfaf34ddd93aac1a12f7d790ea870157f"
Oct 30 20:16:57.349183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.349159    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:57.349282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.349192    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" event={"ID":"cf801a67-90b1-408d-a14b-86907900d29d","Type":"ContainerStarted","Data":"e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc"}
Oct 30 20:16:57.349282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.349222    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" event={"ID":"cf801a67-90b1-408d-a14b-86907900d29d","Type":"ContainerStarted","Data":"b66a735b5865907a4c6bcc31dd59532289af8c510f9b9e170d92f0275accd361"}
Oct 30 20:16:57.349282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.349237    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:57.349393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.349280    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:57.350519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.350500    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:16:57.425932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:57.425878    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podStartSLOduration=2.4258581169999998 podStartE2EDuration="2.425858117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:57.401191261 +0000 UTC m=+750.304611286" watchObservedRunningTime="2026-10-30 20:16:57.425858117 +0000 UTC m=+750.329278141"
Oct 30 20:16:58.203148 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:58.203111    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:16:58.393727 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:58.393684    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" event={"ID":"e4b2a700-f272-4d97-99e7-e894bf75f21f","Type":"ContainerStarted","Data":"8c72767e779cfab71d33cc2751582eaaef0cc4df7927e5dd951df91ab93af159"}
Oct 30 20:16:58.882030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:58.879388    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:16:58.882030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:58.879510    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 20:16:58.882030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:58.879589    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 20:16:59.886573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:59.886525    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 20:16:59.887251 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:16:59.887217    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 20:17:00.882397 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:00.882359    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 20:17:00.882583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:00.882434    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 20:17:01.879978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:01.879944    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 20:17:01.880500 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:01.880467    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 20:17:02.078028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.063178    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"]
Oct 30 20:17:02.084054 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.083551    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.084054 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.083641    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"]
Oct 30 20:17:02.093750 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.093722    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c901b\"/\"kube-root-ca.crt\""
Oct 30 20:17:02.093884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.093810    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c901b\"/\"openshift-service-ca.crt\""
Oct 30 20:17:02.093884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.093855    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c901b\"/\"simulator-w4-9c901b-kserve-self-signed-certs\""
Oct 30 20:17:02.094871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.094846    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c901b\"/\"default-dockercfg-m7fvr\""
Oct 30 20:17:02.132375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.132313    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.132375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.132352    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.132511 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.132450    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.132511 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.132473    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.132511 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.132492    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v7sj\" (UniqueName: \"kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.132635 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.132614    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.233536 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.233496    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.234278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.234253    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.234412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.234314    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.234412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.234338    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.234412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.234365    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v7sj\" (UniqueName: \"kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.234587 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.234447    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.235606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.234198    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.235606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.235555    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.236649 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.236042    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.237985 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.237953    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.258070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.257986    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v7sj\" (UniqueName: \"kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.267025 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.259237    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm\") pod \"simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") " pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.397860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.397770    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:02.698290 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.698268    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"]
Oct 30 20:17:02.700371 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:02.700343    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2d5ac13_acaa_4baf_bfba_c38fb728d8ed.slice/crio-3b364f744d26485b1985ee4f90b929506a06806f71804d99b7519b2c001261ec WatchSource:0}: Error finding container 3b364f744d26485b1985ee4f90b929506a06806f71804d99b7519b2c001261ec: Status 404 returned error can't find the container with id 3b364f744d26485b1985ee4f90b929506a06806f71804d99b7519b2c001261ec
Oct 30 20:17:02.879445 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.879403    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 20:17:02.879708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:02.879678    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 20:17:03.430948 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:03.430910    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" event={"ID":"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed","Type":"ContainerStarted","Data":"b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4"}
Oct 30 20:17:03.431351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:03.430966    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" event={"ID":"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed","Type":"ContainerStarted","Data":"3b364f744d26485b1985ee4f90b929506a06806f71804d99b7519b2c001261ec"}
Oct 30 20:17:03.454767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:03.450980    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podStartSLOduration=1.450966233 podStartE2EDuration="1.450966233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:03.450769335 +0000 UTC m=+756.354189362" watchObservedRunningTime="2026-10-30 20:17:03.450966233 +0000 UTC m=+756.354386258"
Oct 30 20:17:03.884332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:03.884300    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:17:03.884591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:03.884571    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:17:03.885366 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:03.885348    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:17:04.398039 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:04.397985    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:05.229590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:05.229558    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:17:06.203842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:06.203807    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:17:06.220474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:06.220446    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:17:06.473251 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:06.473177    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:17:12.398443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.398397    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:12.412193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.412162    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:12.456810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.456779    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"]
Oct 30 20:17:12.460599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.460570    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.472546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.471591    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"]
Oct 30 20:17:12.472691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.472621    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5197ec\"/\"openshift-service-ca.crt\""
Oct 30 20:17:12.472691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.472657    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5197ec\"/\"premium-w1-5197ec-kserve-self-signed-certs\""
Oct 30 20:17:12.472796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.472714    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5197ec\"/\"kube-root-ca.crt\""
Oct 30 20:17:12.472900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.472875    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5197ec\"/\"default-dockercfg-2nnss\""
Oct 30 20:17:12.499245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.499213    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:17:12.553787 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.553751    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.553976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.553951    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.554086 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.554065    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.554172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.554107    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.554240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.554214    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.554373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.554346    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjql\" (UniqueName: \"kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667094    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjql\" (UniqueName: \"kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667187    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667262    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667325    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667360    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667407    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.667861    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.668460    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.677072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.676609    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.701033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.686463    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.701033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.700669    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjql\" (UniqueName: \"kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.707059 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.706998    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs\") pod \"premium-w1-5197ec-kserve-6fd54756f9-gxx7t\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") " pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:12.792799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:12.792769    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:13.069857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:13.069760    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"]
Oct 30 20:17:13.073745 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:13.073710    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35174ada_bc58_4697_83cb_2d0a4db35dfb.slice/crio-d3acc4d376f76c1c364491bf64f138e792d75dc3f2eaca3e5e7154bf890869d3 WatchSource:0}: Error finding container d3acc4d376f76c1c364491bf64f138e792d75dc3f2eaca3e5e7154bf890869d3: Status 404 returned error can't find the container with id d3acc4d376f76c1c364491bf64f138e792d75dc3f2eaca3e5e7154bf890869d3
Oct 30 20:17:13.492348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:13.492307    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" event={"ID":"35174ada-bc58-4697-83cb-2d0a4db35dfb","Type":"ContainerStarted","Data":"e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126"}
Oct 30 20:17:13.492739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:13.492357    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" event={"ID":"35174ada-bc58-4697-83cb-2d0a4db35dfb","Type":"ContainerStarted","Data":"d3acc4d376f76c1c364491bf64f138e792d75dc3f2eaca3e5e7154bf890869d3"}
Oct 30 20:17:13.514196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:13.514146    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podStartSLOduration=1.514130854 podStartE2EDuration="1.514130854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:13.512975694 +0000 UTC m=+766.416395721" watchObservedRunningTime="2026-10-30 20:17:13.514130854 +0000 UTC m=+766.417550913"
Oct 30 20:17:13.793909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:13.793808    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:16.784644 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:16.784610    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:17:18.587329 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.587293    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"]
Oct 30 20:17:18.591056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.591040    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.601233 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.601212    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-44135b\"/\"openshift-service-ca.crt\""
Oct 30 20:17:18.601355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.601230    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44135b\"/\"premium-w5-44135b-kserve-self-signed-certs\""
Oct 30 20:17:18.601355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.601249    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-44135b\"/\"kube-root-ca.crt\""
Oct 30 20:17:18.601460 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.601436    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44135b\"/\"default-dockercfg-7wx9c\""
Oct 30 20:17:18.603149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.603131    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"]
Oct 30 20:17:18.730853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.730821    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55dg\" (UniqueName: \"kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.730853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.730857    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.731091 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.730888    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.731091 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.730991    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.731174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.731120    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.731174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.731166    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.833635 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.833576    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.833845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.833655    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.833845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.833734    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55dg\" (UniqueName: \"kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.833845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.833761    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.833845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.833807    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.834076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.833884    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.834983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.834951    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.835309 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.835278    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.835504 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.835487    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.837437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.837352    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.837437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.837358    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.847395 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.847369    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55dg\" (UniqueName: \"kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg\") pod \"premium-w5-44135b-kserve-64b677c996-x456z\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") " pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:18.902886 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:18.902850    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:19.181418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:19.181392    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"]
Oct 30 20:17:19.183970 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:19.183940    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eaf0ace_20fd_4aca_a952_13e700efe595.slice/crio-c16e681a2de1556b0cbe81fe65f67fe5d1cc379611c4b20fb14abf8e5109a2c1 WatchSource:0}: Error finding container c16e681a2de1556b0cbe81fe65f67fe5d1cc379611c4b20fb14abf8e5109a2c1: Status 404 returned error can't find the container with id c16e681a2de1556b0cbe81fe65f67fe5d1cc379611c4b20fb14abf8e5109a2c1
Oct 30 20:17:19.528946 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:19.528847    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" event={"ID":"4eaf0ace-20fd-4aca-a952-13e700efe595","Type":"ContainerStarted","Data":"d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5"}
Oct 30 20:17:19.528946 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:19.528896    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" event={"ID":"4eaf0ace-20fd-4aca-a952-13e700efe595","Type":"ContainerStarted","Data":"c16e681a2de1556b0cbe81fe65f67fe5d1cc379611c4b20fb14abf8e5109a2c1"}
Oct 30 20:17:19.549276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:19.549225    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podStartSLOduration=1.549208723 podStartE2EDuration="1.549208723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:19.54912687 +0000 UTC m=+772.452546896" watchObservedRunningTime="2026-10-30 20:17:19.549208723 +0000 UTC m=+772.452628766"
Oct 30 20:17:19.657714 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:17:19.657677    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:17:19.903386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:19.903361    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:22.793344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:22.793307    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:22.807029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:22.806731    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:23.560883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:23.560852    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:17:28.015630 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.015557    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"]
Oct 30 20:17:28.021693 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.021665    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.028754 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.028425    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-909519\"/\"kube-root-ca.crt\""
Oct 30 20:17:28.029617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.029590    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-909519\"/\"default-dockercfg-p2vpb\""
Oct 30 20:17:28.029739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.029675    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-909519\"/\"premium-w2-909519-kserve-self-signed-certs\""
Oct 30 20:17:28.030127 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.030105    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-909519\"/\"openshift-service-ca.crt\""
Oct 30 20:17:28.032303 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.032285    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"]
Oct 30 20:17:28.046927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.046900    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.047047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.046945    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j544w\" (UniqueName: \"kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.047096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.047047    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.047132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.047097    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.047132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.047115    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.047243 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.047224    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148392    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148447    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j544w\" (UniqueName: \"kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148472    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148517    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148545    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148586    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.148944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.148920    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.149189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.149142    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.149189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.149173    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.151931 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.151903    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.152067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.151947    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.162487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.162444    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j544w\" (UniqueName: \"kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w\") pod \"premium-w2-909519-kserve-7ffd9c4cbc-xt2rb\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") " pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.342597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.342520    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:28.623154 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.623051    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"]
Oct 30 20:17:28.626212 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:28.626187    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cabd8f3_70d4_404e_9adb_7589b76bb013.slice/crio-98a2589cfb44e579b30bd7d6993af0b6763553bf7dcef4cbf980b6b44edfd2f8 WatchSource:0}: Error finding container 98a2589cfb44e579b30bd7d6993af0b6763553bf7dcef4cbf980b6b44edfd2f8: Status 404 returned error can't find the container with id 98a2589cfb44e579b30bd7d6993af0b6763553bf7dcef4cbf980b6b44edfd2f8
Oct 30 20:17:28.903195 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.903110    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:28.915957 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:28.915929    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:29.591381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:29.591342    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" event={"ID":"9cabd8f3-70d4-404e-9adb-7589b76bb013","Type":"ContainerStarted","Data":"bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a"}
Oct 30 20:17:29.591876 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:29.591391    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" event={"ID":"9cabd8f3-70d4-404e-9adb-7589b76bb013","Type":"ContainerStarted","Data":"98a2589cfb44e579b30bd7d6993af0b6763553bf7dcef4cbf980b6b44edfd2f8"}
Oct 30 20:17:29.603560 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:29.603526    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:17:29.622496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:29.622455    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podStartSLOduration=2.622442391 podStartE2EDuration="2.622442391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:29.621102488 +0000 UTC m=+782.524522525" watchObservedRunningTime="2026-10-30 20:17:29.622442391 +0000 UTC m=+782.525862416"
Oct 30 20:17:30.343173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:30.343143    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:31.946276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:31.946232    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:17:32.129647 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.129615    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"]
Oct 30 20:17:32.135074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.135048    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.144492 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.144468    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c646da\"/\"default-dockercfg-77sdv\""
Oct 30 20:17:32.145108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.145084    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"]
Oct 30 20:17:32.145213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.145094    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c646da\"/\"premium-w0-c646da-kserve-self-signed-certs\""
Oct 30 20:17:32.146050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.145984    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c646da\"/\"openshift-service-ca.crt\""
Oct 30 20:17:32.146315 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.146290    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c646da\"/\"kube-root-ca.crt\""
Oct 30 20:17:32.195262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.195226    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.195454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.195297    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.195454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.195388    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.195582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.195449    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj6s\" (UniqueName: \"kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.195582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.195533    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.195582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.195567    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.296914 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.296834    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.296914 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.296902    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.297209 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.296928    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.297209 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.296982    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj6s\" (UniqueName: \"kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.297209 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.297071    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.297209 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.297100    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.297431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.297406    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.301041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.298891    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.301041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.297460    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.314080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.314046    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.316081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.316055    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.319382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.316735    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj6s\" (UniqueName: \"kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s\") pod \"premium-w0-c646da-kserve-9bcdbf6f8-99bqh\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") " pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.449651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.449615    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:32.756632 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:32.756599    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"]
Oct 30 20:17:32.760168 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:32.760142    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8cc6c8_8f40_40af_bc28_b3699dab3c88.slice/crio-2297833b08ee41186210f3d57adef4bfe36e778fc60e05940ee11f3b287f5fdd WatchSource:0}: Error finding container 2297833b08ee41186210f3d57adef4bfe36e778fc60e05940ee11f3b287f5fdd: Status 404 returned error can't find the container with id 2297833b08ee41186210f3d57adef4bfe36e778fc60e05940ee11f3b287f5fdd
Oct 30 20:17:33.623257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:33.623221    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" event={"ID":"6d8cc6c8-8f40-40af-bc28-b3699dab3c88","Type":"ContainerStarted","Data":"39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e"}
Oct 30 20:17:33.623257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:33.623264    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" event={"ID":"6d8cc6c8-8f40-40af-bc28-b3699dab3c88","Type":"ContainerStarted","Data":"2297833b08ee41186210f3d57adef4bfe36e778fc60e05940ee11f3b287f5fdd"}
Oct 30 20:17:33.649244 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:33.649194    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podStartSLOduration=1.649181226 podStartE2EDuration="1.649181226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:33.648111592 +0000 UTC m=+786.551531619" watchObservedRunningTime="2026-10-30 20:17:33.649181226 +0000 UTC m=+786.552601251"
Oct 30 20:17:34.449994 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:34.449962    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:38.343653 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:38.343617    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:38.356393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:38.356369    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:38.665410 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:38.665383    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:17:39.123431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.123397    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"]
Oct 30 20:17:39.127467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.127444    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.138041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.138018    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5197ec-openshift-default-dockercfg-fvmqh\""
Oct 30 20:17:39.140436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.140413    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"]
Oct 30 20:17:39.174467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174438    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174487    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174510    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174576    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174680    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174716    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174798    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdjb\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174849    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.174972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.174876    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276309    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276341    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276398    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276423    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276569    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdjb\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276735 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276708    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276787 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276760    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276835 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276813    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276825    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276858    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.276890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.276865    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.277131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.277091    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.277236 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.277219    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.277522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.277505    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.279253 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.279222    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.279455 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.279434    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.290640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.290607    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdjb\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.290887 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.290867    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.441171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.441076    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:39.735895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.735797    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"]
Oct 30 20:17:39.738167 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:39.738142    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bacd98_f21a_4ee5_b0ae_caab7dff3218.slice/crio-3b20984431cdcada18e382dd4b8c55c43db42e5e04c95670539e9c6196e6c014 WatchSource:0}: Error finding container 3b20984431cdcada18e382dd4b8c55c43db42e5e04c95670539e9c6196e6c014: Status 404 returned error can't find the container with id 3b20984431cdcada18e382dd4b8c55c43db42e5e04c95670539e9c6196e6c014
Oct 30 20:17:39.740550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.740445    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:17:39.740633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.740593    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:17:39.740633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:39.740626    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:17:40.687393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:40.687346    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" event={"ID":"f7bacd98-f21a-4ee5-b0ae-caab7dff3218","Type":"ContainerStarted","Data":"7a166363d7cb470dbd8f727718d085d97170d7d86c4cbe4b932d298fca8ed465"}
Oct 30 20:17:40.687928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:40.687421    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" event={"ID":"f7bacd98-f21a-4ee5-b0ae-caab7dff3218","Type":"ContainerStarted","Data":"3b20984431cdcada18e382dd4b8c55c43db42e5e04c95670539e9c6196e6c014"}
Oct 30 20:17:40.714433 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:40.714379    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podStartSLOduration=1.714365156 podStartE2EDuration="1.714365156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:40.713121434 +0000 UTC m=+793.616541461" watchObservedRunningTime="2026-10-30 20:17:40.714365156 +0000 UTC m=+793.617785175"
Oct 30 20:17:41.442276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:41.442240    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:41.443910 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:41.443879    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:17:41.444098 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:41.443936    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:17:42.441756 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:42.441720    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:17:42.442156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:42.441771    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:17:42.450078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:42.450048    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:42.496779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:42.496748    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:42.714224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:42.714143    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:17:43.441891 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:43.441854    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:17:43.442488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:43.441914    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:17:44.442269 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:44.442237    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:17:44.442654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:44.442287    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:17:45.441715 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:45.441678    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:17:45.441893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:45.441738    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:17:47.400478 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:47.400440    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:47.400952 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:47.400826    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:47.401864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:47.401833    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:17:49.697728 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:17:49.696485    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:17:52.117674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.117627    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"]
Oct 30 20:17:52.121964 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.121938    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.130722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.130693    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:17:52.130860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.130765    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:52.132120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.131996    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"]
Oct 30 20:17:52.137590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.137561    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.137708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.137609    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqh46\" (UniqueName: \"kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.137779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.137715    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.200904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.200874    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:17:52.201084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.200957    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:17:52.238897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.238869    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.239051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.238911    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqh46\" (UniqueName: \"kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.239051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.238935    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.239648 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.239630    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.241258 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.241230    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.252099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.252073    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqh46\" (UniqueName: \"kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46\") pod \"maas-api-revoke-sub-81741671-768sp\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.443307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.443221    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:52.727510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.727435    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"]
Oct 30 20:17:52.732023 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:52.731976    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487459d8_9003_44e0_b9b6_1c07b58259e9.slice/crio-ddc78cfba06283e0f4671170c877befb327b8e45112d23a7d6566f33f985858b WatchSource:0}: Error finding container ddc78cfba06283e0f4671170c877befb327b8e45112d23a7d6566f33f985858b: Status 404 returned error can't find the container with id ddc78cfba06283e0f4671170c877befb327b8e45112d23a7d6566f33f985858b
Oct 30 20:17:52.789755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:52.789720    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp" event={"ID":"487459d8-9003-44e0-b9b6-1c07b58259e9","Type":"ContainerStarted","Data":"ddc78cfba06283e0f4671170c877befb327b8e45112d23a7d6566f33f985858b"}
Oct 30 20:17:53.013798 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.013732    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"]
Oct 30 20:17:53.019347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.019325    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.027434 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.026692    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"]
Oct 30 20:17:53.051472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.051441    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.051604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.051474    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h62w\" (UniqueName: \"kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.051604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.051513    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.153021 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.152967    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.153503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.153039    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h62w\" (UniqueName: \"kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.153503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.153083    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.153852 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.153824    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.155575 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.155547    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.166415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.166393    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h62w\" (UniqueName: \"kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w\") pod \"maas-api-revoke-sub-04729ff5-sjmdw\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.334752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.334668    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:53.614056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.614028    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"]
Oct 30 20:17:53.796289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.796262    6058 generic.go:356] "Generic (PLEG): container finished" podID="487459d8-9003-44e0-b9b6-1c07b58259e9" containerID="d1b662d30d6dcd3237ddfce6b97d9fe0d87d9de3ddaae07d227541c1411a3951" exitCode=0
Oct 30 20:17:53.796422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.796358    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp" event={"ID":"487459d8-9003-44e0-b9b6-1c07b58259e9","Type":"ContainerDied","Data":"d1b662d30d6dcd3237ddfce6b97d9fe0d87d9de3ddaae07d227541c1411a3951"}
Oct 30 20:17:53.797644 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:53.797620    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw" event={"ID":"2adeadd1-9091-4900-8e85-4b8d0ad98238","Type":"ContainerStarted","Data":"193a6641d3949436582ee7366ade1fa578615cb04a768947dec31f704b8d58ec"}
Oct 30 20:17:54.244948 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.244863    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"]
Oct 30 20:17:54.249155 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.249130    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.256203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.256175    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-124fd1-openshift-default-dockercfg-n5plt\""
Oct 30 20:17:54.265820 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.265780    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"]
Oct 30 20:17:54.367158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367120    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367170    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367273    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367317    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367361    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367457    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cnxc\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367501    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367535    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.367749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.367587    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469066 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469031    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469088    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469131    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469170    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469216    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469273    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cnxc\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469321    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469358    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469415    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469485    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469546    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469884    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.469996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.469923    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.470115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.470088    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.471669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.471644    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.472053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.472033    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.482775 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.482748    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.482985 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.482961    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cnxc\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc\") pod \"e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") " pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.565280 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.565194    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:54.804352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.804317    6058 generic.go:356] "Generic (PLEG): container finished" podID="2adeadd1-9091-4900-8e85-4b8d0ad98238" containerID="8ceb035aa3ff5bb20203493e46a2194a01c305efe057062875ab604e099c6853" exitCode=0
Oct 30 20:17:54.804532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.804365    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw" event={"ID":"2adeadd1-9091-4900-8e85-4b8d0ad98238","Type":"ContainerDied","Data":"8ceb035aa3ff5bb20203493e46a2194a01c305efe057062875ab604e099c6853"}
Oct 30 20:17:54.853020 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.852919    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"]
Oct 30 20:17:54.858565 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:17:54.858532    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5283e6_a3a3_43ea_bc08_08cfbadd65ef.slice/crio-11c436b849cf9d18426f924d8681d039ec44f1ccb1214b236deb08503506c305 WatchSource:0}: Error finding container 11c436b849cf9d18426f924d8681d039ec44f1ccb1214b236deb08503506c305: Status 404 returned error can't find the container with id 11c436b849cf9d18426f924d8681d039ec44f1ccb1214b236deb08503506c305
Oct 30 20:17:54.872081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.872040    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:17:54.872177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.872136    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:17:54.872229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.872186    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:17:54.942231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.942203    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:54.980394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.980368    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46\" (UniqueName: \"kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46\") pod \"487459d8-9003-44e0-b9b6-1c07b58259e9\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") "
Oct 30 20:17:54.983132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.980457    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca\") pod \"487459d8-9003-44e0-b9b6-1c07b58259e9\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") "
Oct 30 20:17:54.983132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.980584    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca\") pod \"487459d8-9003-44e0-b9b6-1c07b58259e9\" (UID: \"487459d8-9003-44e0-b9b6-1c07b58259e9\") "
Oct 30 20:17:54.983132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.980906    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca" pod "487459d8-9003-44e0-b9b6-1c07b58259e9" (UID: "487459d8-9003-44e0-b9b6-1c07b58259e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:54.983132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.981065    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:54.983132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.982576    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca" pod "487459d8-9003-44e0-b9b6-1c07b58259e9" (UID: "487459d8-9003-44e0-b9b6-1c07b58259e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:54.985053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:54.984976    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46" pod "487459d8-9003-44e0-b9b6-1c07b58259e9" (UID: "487459d8-9003-44e0-b9b6-1c07b58259e9"). InnerVolumeSpecName "kube-api-access-rqh46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:55.081961 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.081920    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487459d8-9003-44e0-b9b6-1c07b58259e9-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:55.081961 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.081952    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqh46\" (UniqueName: \"kubernetes.io/projected/487459d8-9003-44e0-b9b6-1c07b58259e9-kube-api-access-rqh46\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:55.812164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.812119    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp" event={"ID":"487459d8-9003-44e0-b9b6-1c07b58259e9","Type":"ContainerDied","Data":"ddc78cfba06283e0f4671170c877befb327b8e45112d23a7d6566f33f985858b"}
Oct 30 20:17:55.812646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.812184    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddc78cfba06283e0f4671170c877befb327b8e45112d23a7d6566f33f985858b"
Oct 30 20:17:55.812646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.812252    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"
Oct 30 20:17:55.822768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.816243    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" event={"ID":"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef","Type":"ContainerStarted","Data":"fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf"}
Oct 30 20:17:55.822768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.816293    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" event={"ID":"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef","Type":"ContainerStarted","Data":"11c436b849cf9d18426f924d8681d039ec44f1ccb1214b236deb08503506c305"}
Oct 30 20:17:55.879530 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:55.876628    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podStartSLOduration=1.876603692 podStartE2EDuration="1.876603692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:55.856475671 +0000 UTC m=+808.759895698" watchObservedRunningTime="2026-10-30 20:17:55.876603692 +0000 UTC m=+808.780023718"
Oct 30 20:17:56.065210 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.065125    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:56.096389 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.096354    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w\" (UniqueName: \"kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w\") pod \"2adeadd1-9091-4900-8e85-4b8d0ad98238\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") "
Oct 30 20:17:56.096570 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.096420    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca\") pod \"2adeadd1-9091-4900-8e85-4b8d0ad98238\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") "
Oct 30 20:17:56.096570 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.096457    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca\") pod \"2adeadd1-9091-4900-8e85-4b8d0ad98238\" (UID: \"2adeadd1-9091-4900-8e85-4b8d0ad98238\") "
Oct 30 20:17:56.097151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.097123    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca" pod "2adeadd1-9091-4900-8e85-4b8d0ad98238" (UID: "2adeadd1-9091-4900-8e85-4b8d0ad98238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:56.099238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.099209    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w" pod "2adeadd1-9091-4900-8e85-4b8d0ad98238" (UID: "2adeadd1-9091-4900-8e85-4b8d0ad98238"). InnerVolumeSpecName "kube-api-access-2h62w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:56.099347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.099276    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca" pod "2adeadd1-9091-4900-8e85-4b8d0ad98238" (UID: "2adeadd1-9091-4900-8e85-4b8d0ad98238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:56.198405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.198372    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h62w\" (UniqueName: \"kubernetes.io/projected/2adeadd1-9091-4900-8e85-4b8d0ad98238-kube-api-access-2h62w\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.198405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.198401    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.198405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.198413    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeadd1-9091-4900-8e85-4b8d0ad98238-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.569405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.569057    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:17:56.570669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.570637    6058 patch_prober.go:29] interesting pod/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:17:56.570885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.570860    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:17:56.822048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.821952    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"
Oct 30 20:17:56.822048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.821985    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw" event={"ID":"2adeadd1-9091-4900-8e85-4b8d0ad98238","Type":"ContainerDied","Data":"193a6641d3949436582ee7366ade1fa578615cb04a768947dec31f704b8d58ec"}
Oct 30 20:17:56.822577 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.822100    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="193a6641d3949436582ee7366ade1fa578615cb04a768947dec31f704b8d58ec"
Oct 30 20:17:56.981028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:56.977294    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:17:57.570028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:57.569353    6058 patch_prober.go:29] interesting pod/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:17:57.570028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:57.569416    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:17:58.569027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:58.568348    6058 patch_prober.go:29] interesting pod/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:17:58.569027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:58.568409    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:17:59.567029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:59.566470    6058 patch_prober.go:29] interesting pod/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:17:59.567029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:17:59.566538    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:18:00.570028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.567326    6058 patch_prober.go:29] interesting pod/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:18:00.570028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.567386    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.962627    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"]
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.963435    6058 cpu_manager.go:414] "containerMap: removing container" podUID="487459d8-9003-44e0-b9b6-1c07b58259e9" containerName="revoke-keys"
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.963458    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487459d8-9003-44e0-b9b6-1c07b58259e9" containerName="revoke-keys"
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.963494    6058 cpu_manager.go:414] "containerMap: removing container" podUID="2adeadd1-9091-4900-8e85-4b8d0ad98238" containerName="revoke-keys"
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.963501    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adeadd1-9091-4900-8e85-4b8d0ad98238" containerName="revoke-keys"
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.963657    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="487459d8-9003-44e0-b9b6-1c07b58259e9" containerName="revoke-keys"
Oct 30 20:18:00.973028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.963673    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adeadd1-9091-4900-8e85-4b8d0ad98238" containerName="revoke-keys"
Oct 30 20:18:00.977036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.976183    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"]
Oct 30 20:18:00.977036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.976295    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:00.985382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.985354    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:00.986998 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.986970    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:18:00.987140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.987073    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:00.987140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:00.987103    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:01.098379 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.098336    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.098535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.098410    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbww\" (UniqueName: \"kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.098535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.098446    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.201878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.201821    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.202092 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.201905    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbww\" (UniqueName: \"kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.202092 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.201938    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.203928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.203883    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.204870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.204843    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.217630 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.217602    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbww\" (UniqueName: \"kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww\") pod \"maas-api-revoke-sub-ad7a3060-8zgrx\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.324479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.324390    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:01.565865 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.565824    6058 patch_prober.go:29] interesting pod/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:18:01.566077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.565884    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:18:01.637639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.637609    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"]
Oct 30 20:18:01.638445 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:01.638417    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2172cdc_054e_4e01_9670_3b2bacb46b13.slice/crio-86add1c2b3b23e0abecb432a3e98671084cde9d410a9e86f4db651e9af4c648a WatchSource:0}: Error finding container 86add1c2b3b23e0abecb432a3e98671084cde9d410a9e86f4db651e9af4c648a: Status 404 returned error can't find the container with id 86add1c2b3b23e0abecb432a3e98671084cde9d410a9e86f4db651e9af4c648a
Oct 30 20:18:01.857895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:01.857858    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx" event={"ID":"b2172cdc-054e-4e01-9670-3b2bacb46b13","Type":"ContainerStarted","Data":"86add1c2b3b23e0abecb432a3e98671084cde9d410a9e86f4db651e9af4c648a"}
Oct 30 20:18:02.576609 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:02.576578    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:18:02.577058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:02.576998    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:18:02.578252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:02.578231    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:18:02.866761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:02.866718    6058 generic.go:356] "Generic (PLEG): container finished" podID="b2172cdc-054e-4e01-9670-3b2bacb46b13" containerID="1868719fdeb2d2ffd26722c8b7df5541264b0cbcf03d30a0ab9231b80d9cb926" exitCode=0
Oct 30 20:18:02.867895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:02.867865    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx" event={"ID":"b2172cdc-054e-4e01-9670-3b2bacb46b13","Type":"ContainerDied","Data":"1868719fdeb2d2ffd26722c8b7df5541264b0cbcf03d30a0ab9231b80d9cb926"}
Oct 30 20:18:03.081749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.081708    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"]
Oct 30 20:18:03.085620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.085598    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.097508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.097482    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"]
Oct 30 20:18:03.248245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.248176    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6879\" (UniqueName: \"kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.248245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.248224    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.248424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.248378    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.349716 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.349674    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6879\" (UniqueName: \"kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.349904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.349743    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.349904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.349822    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.350453 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.350429    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.352432 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.352407    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.363496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.363467    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6879\" (UniqueName: \"kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879\") pod \"maas-api-revoke-sub-cef31f55-dh6v2\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.403518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.403481    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:03.701726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.701702    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"]
Oct 30 20:18:03.704691 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:03.704659    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2dfc3a_4fbc_4714_b988_312e75702733.slice/crio-b75f299613bb6a1d19fb430d0a5e59e8522e6cc96ba244c2feff6c888d27cc69 WatchSource:0}: Error finding container b75f299613bb6a1d19fb430d0a5e59e8522e6cc96ba244c2feff6c888d27cc69: Status 404 returned error can't find the container with id b75f299613bb6a1d19fb430d0a5e59e8522e6cc96ba244c2feff6c888d27cc69
Oct 30 20:18:03.885312 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:03.885273    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2" event={"ID":"fe2dfc3a-4fbc-4714-b988-312e75702733","Type":"ContainerStarted","Data":"b75f299613bb6a1d19fb430d0a5e59e8522e6cc96ba244c2feff6c888d27cc69"}
Oct 30 20:18:04.050075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.050051    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:04.183029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.182217    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca\") pod \"b2172cdc-054e-4e01-9670-3b2bacb46b13\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") "
Oct 30 20:18:04.183029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.182313    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca\") pod \"b2172cdc-054e-4e01-9670-3b2bacb46b13\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") "
Oct 30 20:18:04.183029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.182434    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww\" (UniqueName: \"kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww\") pod \"b2172cdc-054e-4e01-9670-3b2bacb46b13\" (UID: \"b2172cdc-054e-4e01-9670-3b2bacb46b13\") "
Oct 30 20:18:04.183029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.182994    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca" pod "b2172cdc-054e-4e01-9670-3b2bacb46b13" (UID: "b2172cdc-054e-4e01-9670-3b2bacb46b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:04.185495 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.185465    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww" pod "b2172cdc-054e-4e01-9670-3b2bacb46b13" (UID: "b2172cdc-054e-4e01-9670-3b2bacb46b13"). InnerVolumeSpecName "kube-api-access-4wbww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:04.185708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.185683    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca" pod "b2172cdc-054e-4e01-9670-3b2bacb46b13" (UID: "b2172cdc-054e-4e01-9670-3b2bacb46b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:04.283953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.283858    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:04.283953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.283893    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbww\" (UniqueName: \"kubernetes.io/projected/b2172cdc-054e-4e01-9670-3b2bacb46b13-kube-api-access-4wbww\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:04.283953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.283903    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2172cdc-054e-4e01-9670-3b2bacb46b13-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:04.868321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.868279    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"]
Oct 30 20:18:04.869176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.869153    6058 cpu_manager.go:414] "containerMap: removing container" podUID="b2172cdc-054e-4e01-9670-3b2bacb46b13" containerName="revoke-keys"
Oct 30 20:18:04.869228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.869180    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2172cdc-054e-4e01-9670-3b2bacb46b13" containerName="revoke-keys"
Oct 30 20:18:04.869299 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.869288    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2172cdc-054e-4e01-9670-3b2bacb46b13" containerName="revoke-keys"
Oct 30 20:18:04.873175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.873148    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:04.916579 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.914696    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"]
Oct 30 20:18:04.920187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.920101    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"
Oct 30 20:18:04.920370 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.920119    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx" event={"ID":"b2172cdc-054e-4e01-9670-3b2bacb46b13","Type":"ContainerDied","Data":"86add1c2b3b23e0abecb432a3e98671084cde9d410a9e86f4db651e9af4c648a"}
Oct 30 20:18:04.920541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.920523    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86add1c2b3b23e0abecb432a3e98671084cde9d410a9e86f4db651e9af4c648a"
Oct 30 20:18:04.922023 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.921990    6058 generic.go:356] "Generic (PLEG): container finished" podID="fe2dfc3a-4fbc-4714-b988-312e75702733" containerID="688b84f88a0d565691ce979d8be3f9a947f28e319662a531266570f97f10a3a1" exitCode=0
Oct 30 20:18:04.922171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:04.922084    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2" event={"ID":"fe2dfc3a-4fbc-4714-b988-312e75702733","Type":"ContainerDied","Data":"688b84f88a0d565691ce979d8be3f9a947f28e319662a531266570f97f10a3a1"}
Oct 30 20:18:05.005749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.005710    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.005947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.005896    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrcvf\" (UniqueName: \"kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.005947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.005923    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.109034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.108770    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrcvf\" (UniqueName: \"kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.109034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.108820    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.109034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.108935    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.111210 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.110092    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.114085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.114036    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.134022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.127606    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrcvf\" (UniqueName: \"kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf\") pod \"maas-api-revoke-sub-bc4cc090-xbbpj\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.240581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.226242    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:05.624695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.624660    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"]
Oct 30 20:18:05.929869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:05.929778    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj" event={"ID":"cde00931-8b7e-4a35-9513-ada054139ac3","Type":"ContainerStarted","Data":"e1ee6b7b059d1860ae5093b572c203dca0970680d097c2e258d5db1c76af5dec"}
Oct 30 20:18:06.082020 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.081988    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:06.166701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.166671    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca\") pod \"fe2dfc3a-4fbc-4714-b988-312e75702733\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") "
Oct 30 20:18:06.166858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.166735    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879\" (UniqueName: \"kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879\") pod \"fe2dfc3a-4fbc-4714-b988-312e75702733\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") "
Oct 30 20:18:06.166858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.166813    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca\") pod \"fe2dfc3a-4fbc-4714-b988-312e75702733\" (UID: \"fe2dfc3a-4fbc-4714-b988-312e75702733\") "
Oct 30 20:18:06.167261 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.167227    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca" pod "fe2dfc3a-4fbc-4714-b988-312e75702733" (UID: "fe2dfc3a-4fbc-4714-b988-312e75702733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:06.168897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.168876    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca" pod "fe2dfc3a-4fbc-4714-b988-312e75702733" (UID: "fe2dfc3a-4fbc-4714-b988-312e75702733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:06.168978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.168913    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879" pod "fe2dfc3a-4fbc-4714-b988-312e75702733" (UID: "fe2dfc3a-4fbc-4714-b988-312e75702733"). InnerVolumeSpecName "kube-api-access-j6879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:06.268392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.268307    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6879\" (UniqueName: \"kubernetes.io/projected/fe2dfc3a-4fbc-4714-b988-312e75702733-kube-api-access-j6879\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:06.268392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.268332    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:06.268392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.268349    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe2dfc3a-4fbc-4714-b988-312e75702733-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:06.358347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.358313    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"]
Oct 30 20:18:06.358901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.358880    6058 cpu_manager.go:414] "containerMap: removing container" podUID="fe2dfc3a-4fbc-4714-b988-312e75702733" containerName="revoke-keys"
Oct 30 20:18:06.358901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.358901    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2dfc3a-4fbc-4714-b988-312e75702733" containerName="revoke-keys"
Oct 30 20:18:06.360162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.360129    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2dfc3a-4fbc-4714-b988-312e75702733" containerName="revoke-keys"
Oct 30 20:18:06.364995 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.364254    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.380804 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.380767    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"]
Oct 30 20:18:06.473907 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.473867    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.474069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.473938    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.474137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.474114    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txnmv\" (UniqueName: \"kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.575404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.575367    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txnmv\" (UniqueName: \"kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.575544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.575447    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.575544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.575477    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.576353 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.576326    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.577928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.577896    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.593631 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.593603    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txnmv\" (UniqueName: \"kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv\") pod \"maas-api-revoke-sub-f2770f9b-vwwhb\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.698197 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.698149    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:06.936803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.936766    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2" event={"ID":"fe2dfc3a-4fbc-4714-b988-312e75702733","Type":"ContainerDied","Data":"b75f299613bb6a1d19fb430d0a5e59e8522e6cc96ba244c2feff6c888d27cc69"}
Oct 30 20:18:06.937312 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.936848    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b75f299613bb6a1d19fb430d0a5e59e8522e6cc96ba244c2feff6c888d27cc69"
Oct 30 20:18:06.937312 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.936782    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"
Oct 30 20:18:06.938485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.938457    6058 generic.go:356] "Generic (PLEG): container finished" podID="cde00931-8b7e-4a35-9513-ada054139ac3" containerID="a790555e46dd090a1fc85ef836cb261288f3881284a6c4b1416c5ad23bb68765" exitCode=0
Oct 30 20:18:06.938597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:06.938492    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj" event={"ID":"cde00931-8b7e-4a35-9513-ada054139ac3","Type":"ContainerDied","Data":"a790555e46dd090a1fc85ef836cb261288f3881284a6c4b1416c5ad23bb68765"}
Oct 30 20:18:07.045423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:07.045400    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"]
Oct 30 20:18:07.047466 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:07.047439    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff69aec_5262_4f87_a45f_25b8ac8988a3.slice/crio-cbc4c88211696b1e9f63dda8380bb449069b2362c7f3c851190bf67a4031134b WatchSource:0}: Error finding container cbc4c88211696b1e9f63dda8380bb449069b2362c7f3c851190bf67a4031134b: Status 404 returned error can't find the container with id cbc4c88211696b1e9f63dda8380bb449069b2362c7f3c851190bf67a4031134b
Oct 30 20:18:07.945521 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:07.945486    6058 generic.go:356] "Generic (PLEG): container finished" podID="6ff69aec-5262-4f87-a45f-25b8ac8988a3" containerID="f93f046fab3f7b99fdb20e224dbe3ff0b392a062d9ba64910dfcef55a6149a29" exitCode=0
Oct 30 20:18:07.945892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:07.945569    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb" event={"ID":"6ff69aec-5262-4f87-a45f-25b8ac8988a3","Type":"ContainerDied","Data":"f93f046fab3f7b99fdb20e224dbe3ff0b392a062d9ba64910dfcef55a6149a29"}
Oct 30 20:18:07.945892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:07.945631    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb" event={"ID":"6ff69aec-5262-4f87-a45f-25b8ac8988a3","Type":"ContainerStarted","Data":"cbc4c88211696b1e9f63dda8380bb449069b2362c7f3c851190bf67a4031134b"}
Oct 30 20:18:08.114216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.114189    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:08.197518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.197440    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf\" (UniqueName: \"kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf\") pod \"cde00931-8b7e-4a35-9513-ada054139ac3\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") "
Oct 30 20:18:08.197674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.197566    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca\") pod \"cde00931-8b7e-4a35-9513-ada054139ac3\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") "
Oct 30 20:18:08.197733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.197714    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca\") pod \"cde00931-8b7e-4a35-9513-ada054139ac3\" (UID: \"cde00931-8b7e-4a35-9513-ada054139ac3\") "
Oct 30 20:18:08.198087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.198060    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca" pod "cde00931-8b7e-4a35-9513-ada054139ac3" (UID: "cde00931-8b7e-4a35-9513-ada054139ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:08.198207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.198159    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:08.199655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.199633    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca" pod "cde00931-8b7e-4a35-9513-ada054139ac3" (UID: "cde00931-8b7e-4a35-9513-ada054139ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:08.200030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.199996    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf" pod "cde00931-8b7e-4a35-9513-ada054139ac3" (UID: "cde00931-8b7e-4a35-9513-ada054139ac3"). InnerVolumeSpecName "kube-api-access-zrcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:08.299287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.299142    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrcvf\" (UniqueName: \"kubernetes.io/projected/cde00931-8b7e-4a35-9513-ada054139ac3-kube-api-access-zrcvf\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:08.299287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.299174    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde00931-8b7e-4a35-9513-ada054139ac3-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:08.466881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.466813    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"]
Oct 30 20:18:08.467302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.467287    6058 cpu_manager.go:414] "containerMap: removing container" podUID="cde00931-8b7e-4a35-9513-ada054139ac3" containerName="revoke-keys"
Oct 30 20:18:08.467351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.467303    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde00931-8b7e-4a35-9513-ada054139ac3" containerName="revoke-keys"
Oct 30 20:18:08.467408 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.467399    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde00931-8b7e-4a35-9513-ada054139ac3" containerName="revoke-keys"
Oct 30 20:18:08.470675 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.470659    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.481160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.480663    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"]
Oct 30 20:18:08.501367 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.501328    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.501494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.501374    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc87j\" (UniqueName: \"kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.501593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.501570    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.602819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.602788    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.602819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.602821    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc87j\" (UniqueName: \"kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.603033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.602946    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.603603 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.603581    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.605204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.605183    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.615988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.615940    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc87j\" (UniqueName: \"kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j\") pod \"maas-api-revoke-sub-75c882e2-f4mnk\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.789274 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.789184    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:08.952412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.952381    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"
Oct 30 20:18:08.952824 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.952413    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj" event={"ID":"cde00931-8b7e-4a35-9513-ada054139ac3","Type":"ContainerDied","Data":"e1ee6b7b059d1860ae5093b572c203dca0970680d097c2e258d5db1c76af5dec"}
Oct 30 20:18:08.952824 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:08.952471    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ee6b7b059d1860ae5093b572c203dca0970680d097c2e258d5db1c76af5dec"
Oct 30 20:18:09.067868 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.067842    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"]
Oct 30 20:18:09.070547 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:09.070518    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d8dc50_9f1c_4b3b_9bcc_7327aeb7046f.slice/crio-0b5f74f14add90134a37bc20e2ab16d627d30cdc26a9fdfeb5ab837ba220c2ad WatchSource:0}: Error finding container 0b5f74f14add90134a37bc20e2ab16d627d30cdc26a9fdfeb5ab837ba220c2ad: Status 404 returned error can't find the container with id 0b5f74f14add90134a37bc20e2ab16d627d30cdc26a9fdfeb5ab837ba220c2ad
Oct 30 20:18:09.183094 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.183066    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:09.209782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.209757    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv\" (UniqueName: \"kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv\") pod \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") "
Oct 30 20:18:09.209986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.209970    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca\") pod \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") "
Oct 30 20:18:09.210061 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.210044    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca\") pod \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\" (UID: \"6ff69aec-5262-4f87-a45f-25b8ac8988a3\") "
Oct 30 20:18:09.210345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.210322    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca" pod "6ff69aec-5262-4f87-a45f-25b8ac8988a3" (UID: "6ff69aec-5262-4f87-a45f-25b8ac8988a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:09.210532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.210492    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:09.212411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.212372    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca" pod "6ff69aec-5262-4f87-a45f-25b8ac8988a3" (UID: "6ff69aec-5262-4f87-a45f-25b8ac8988a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:09.212526 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.212477    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv" pod "6ff69aec-5262-4f87-a45f-25b8ac8988a3" (UID: "6ff69aec-5262-4f87-a45f-25b8ac8988a3"). InnerVolumeSpecName "kube-api-access-txnmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:09.311384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.311353    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txnmv\" (UniqueName: \"kubernetes.io/projected/6ff69aec-5262-4f87-a45f-25b8ac8988a3-kube-api-access-txnmv\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:09.311384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.311383    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff69aec-5262-4f87-a45f-25b8ac8988a3-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:09.652552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.652463    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"]
Oct 30 20:18:09.653730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.653699    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6ff69aec-5262-4f87-a45f-25b8ac8988a3" containerName="revoke-keys"
Oct 30 20:18:09.653730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.653728    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff69aec-5262-4f87-a45f-25b8ac8988a3" containerName="revoke-keys"
Oct 30 20:18:09.653980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.653965    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff69aec-5262-4f87-a45f-25b8ac8988a3" containerName="revoke-keys"
Oct 30 20:18:09.661109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.661082    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.676504 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.676473    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"]
Oct 30 20:18:09.715409 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715377    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715592 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715418    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715592 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715445    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715592 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715583    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715775 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715720    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715834 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715800    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715834 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715821    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715838    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzq5r\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.715937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.715857    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.740368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.740330    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9c901b-openshift-default-dockercfg-hnkkz\""
Oct 30 20:18:09.816885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.816852    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.816924    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.816969    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817037    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817055    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817070    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzq5r\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817095    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817150    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817186    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817438 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817406    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817463    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817480    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817715    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.817855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.817823    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.819622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.819597    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.819801 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.819776    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.829529 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.829499    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.829711 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.829691    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzq5r\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.961609 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.961533    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"
Oct 30 20:18:09.962063 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.961541    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb" event={"ID":"6ff69aec-5262-4f87-a45f-25b8ac8988a3","Type":"ContainerDied","Data":"cbc4c88211696b1e9f63dda8380bb449069b2362c7f3c851190bf67a4031134b"}
Oct 30 20:18:09.962063 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.961684    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbc4c88211696b1e9f63dda8380bb449069b2362c7f3c851190bf67a4031134b"
Oct 30 20:18:09.963569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.963515    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk" event={"ID":"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f","Type":"ContainerStarted","Data":"a67a3bb56caf5b32bb161a2940570caf572ea7f8e81a4df623cdad0877d2e9ae"}
Oct 30 20:18:09.963569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.963567    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk" event={"ID":"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f","Type":"ContainerStarted","Data":"0b5f74f14add90134a37bc20e2ab16d627d30cdc26a9fdfeb5ab837ba220c2ad"}
Oct 30 20:18:09.977262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.977237    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:09.984923 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:09.984879    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk" podStartSLOduration=1.984867063 podStartE2EDuration="1.984867063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:09.982613675 +0000 UTC m=+822.886033702" watchObservedRunningTime="2026-10-30 20:18:09.984867063 +0000 UTC m=+822.888287136"
Oct 30 20:18:10.262133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.262109    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"]
Oct 30 20:18:10.263795 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:10.263759    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35a3764_7e24_4911_bffa_65ad0c60cdf3.slice/crio-b00d727f20994afb5bda8fb14b1c9fca750e0747d77c4324d110c33a4b27acea WatchSource:0}: Error finding container b00d727f20994afb5bda8fb14b1c9fca750e0747d77c4324d110c33a4b27acea: Status 404 returned error can't find the container with id b00d727f20994afb5bda8fb14b1c9fca750e0747d77c4324d110c33a4b27acea
Oct 30 20:18:10.266133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.266097    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:10.266229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.266174    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:10.266229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.266216    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:10.985743 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.985701    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" event={"ID":"c35a3764-7e24-4911-bffa-65ad0c60cdf3","Type":"ContainerStarted","Data":"389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889"}
Oct 30 20:18:10.986249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.985769    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" event={"ID":"c35a3764-7e24-4911-bffa-65ad0c60cdf3","Type":"ContainerStarted","Data":"b00d727f20994afb5bda8fb14b1c9fca750e0747d77c4324d110c33a4b27acea"}
Oct 30 20:18:10.987533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.987501    6058 generic.go:356] "Generic (PLEG): container finished" podID="b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" containerID="a67a3bb56caf5b32bb161a2940570caf572ea7f8e81a4df623cdad0877d2e9ae" exitCode=0
Oct 30 20:18:10.987669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:10.987556    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk" event={"ID":"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f","Type":"ContainerDied","Data":"a67a3bb56caf5b32bb161a2940570caf572ea7f8e81a4df623cdad0877d2e9ae"}
Oct 30 20:18:11.037069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:11.036998    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podStartSLOduration=2.036982079 podStartE2EDuration="2.036982079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:11.013649942 +0000 UTC m=+823.917069967" watchObservedRunningTime="2026-10-30 20:18:11.036982079 +0000 UTC m=+823.940402106"
Oct 30 20:18:11.978096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:11.978041    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:11.979734 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:11.979695    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:18:11.979896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:11.979758    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:18:12.185817 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.185788    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:12.249519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.249436    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca\") pod \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") "
Oct 30 20:18:12.249684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.249528    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j\" (UniqueName: \"kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j\") pod \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") "
Oct 30 20:18:12.249684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.249574    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca\") pod \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\" (UID: \"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f\") "
Oct 30 20:18:12.249996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.249971    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca" pod "b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" (UID: "b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:12.250186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.250165    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:12.265037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.261455    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca" pod "b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" (UID: "b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:12.265037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.261909    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j" pod "b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" (UID: "b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f"). InnerVolumeSpecName "kube-api-access-lc87j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:12.355204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.351771    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:12.355204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.351816    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc87j\" (UniqueName: \"kubernetes.io/projected/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f-kube-api-access-lc87j\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:12.982091 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.981495    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:18:12.982091 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:12.981559    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:18:13.001828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:13.001793    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"
Oct 30 20:18:13.001986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:13.001828    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk" event={"ID":"b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f","Type":"ContainerDied","Data":"0b5f74f14add90134a37bc20e2ab16d627d30cdc26a9fdfeb5ab837ba220c2ad"}
Oct 30 20:18:13.001986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:13.001887    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5f74f14add90134a37bc20e2ab16d627d30cdc26a9fdfeb5ab837ba220c2ad"
Oct 30 20:18:13.978733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:13.978694    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:18:13.979106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:13.978777    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:18:14.978482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:14.978441    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:18:14.978806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:14.978515    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:18:15.391169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.391135    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"]
Oct 30 20:18:15.391716 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.391695    6058 cpu_manager.go:414] "containerMap: removing container" podUID="b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" containerName="revoke-keys"
Oct 30 20:18:15.391813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.391718    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" containerName="revoke-keys"
Oct 30 20:18:15.391936 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.391853    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" containerName="revoke-keys"
Oct 30 20:18:15.394854 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.394834    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.404274 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.404241    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:15.404387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.404281    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:15.404451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.404423    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"]
Oct 30 20:18:15.404696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.404673    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:15.404696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.404681    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:18:15.527402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.527106    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.527402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.527252    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.527402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.527327    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvj9\" (UniqueName: \"kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.628680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.628646    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.628890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.628762    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.628890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.628831    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvj9\" (UniqueName: \"kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.629408 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.629381    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.631133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.631111    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.641996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.641939    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvj9\" (UniqueName: \"kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9\") pod \"maas-api-revoke-sub-5fd7bc4d-pc5vf\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.710320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.710288    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:15.982411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.982337    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:18:15.982411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:15.982386    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:18:16.037566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.036838    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"]
Oct 30 20:18:16.044040 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:16.040210    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad047721_5c69_4d66_b297_6afe97b8b803.slice/crio-b243a6be59ee91b28717a8fd55c6d3c2d5f8ab91037504f01809c7fb6a1d41bc WatchSource:0}: Error finding container b243a6be59ee91b28717a8fd55c6d3c2d5f8ab91037504f01809c7fb6a1d41bc: Status 404 returned error can't find the container with id b243a6be59ee91b28717a8fd55c6d3c2d5f8ab91037504f01809c7fb6a1d41bc
Oct 30 20:18:16.820322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.820288    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"]
Oct 30 20:18:16.827477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.827445    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:16.835712 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.835335    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"]
Oct 30 20:18:16.943019 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.942968    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rfl\" (UniqueName: \"kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:16.943193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.943145    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:16.943258 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:16.943209    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.028859 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.028822    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf" event={"ID":"ad047721-5c69-4d66-b297-6afe97b8b803","Type":"ContainerStarted","Data":"d56b8886447abc9217eb497cd78afebddf3d91e170f6b6536c1b87ea83c8e03e"}
Oct 30 20:18:17.029347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.028877    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf" event={"ID":"ad047721-5c69-4d66-b297-6afe97b8b803","Type":"ContainerStarted","Data":"b243a6be59ee91b28717a8fd55c6d3c2d5f8ab91037504f01809c7fb6a1d41bc"}
Oct 30 20:18:17.044082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.044050    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.044257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.044148    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.044332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.044282    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rfl\" (UniqueName: \"kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.045404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.045371    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.050606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.050581    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.059632 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.059602    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rfl\" (UniqueName: \"kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl\") pod \"maas-api-revoke-sub-c55b52bf-bn5vs\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.061176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.061125    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf" podStartSLOduration=2.061109613 podStartE2EDuration="2.061109613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:17.058410501 +0000 UTC m=+829.961830528" watchObservedRunningTime="2026-10-30 20:18:17.061109613 +0000 UTC m=+829.964529639"
Oct 30 20:18:17.068162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.068132    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:17.070708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.068599    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:17.070708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.069744    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:18:17.154596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.154547    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:17.447945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:17.447921    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"]
Oct 30 20:18:17.449524 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:17.449496    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582fd540_84d2_4fe7_aa89_e8e5212710c8.slice/crio-14bc44b9401f84cc9e85fd873e5cf076a2152bb3cd94c9b25f14a99be1f7e181 WatchSource:0}: Error finding container 14bc44b9401f84cc9e85fd873e5cf076a2152bb3cd94c9b25f14a99be1f7e181: Status 404 returned error can't find the container with id 14bc44b9401f84cc9e85fd873e5cf076a2152bb3cd94c9b25f14a99be1f7e181
Oct 30 20:18:18.039254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:18.038925    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs" event={"ID":"582fd540-84d2-4fe7-aa89-e8e5212710c8","Type":"ContainerStarted","Data":"4cdce80d94d7b2141f71f36c158e9089fd6c97e09a76f0750d1961229647a853"}
Oct 30 20:18:18.039254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:18.039000    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs" event={"ID":"582fd540-84d2-4fe7-aa89-e8e5212710c8","Type":"ContainerStarted","Data":"14bc44b9401f84cc9e85fd873e5cf076a2152bb3cd94c9b25f14a99be1f7e181"}
Oct 30 20:18:18.041363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:18.041322    6058 generic.go:356] "Generic (PLEG): container finished" podID="ad047721-5c69-4d66-b297-6afe97b8b803" containerID="d56b8886447abc9217eb497cd78afebddf3d91e170f6b6536c1b87ea83c8e03e" exitCode=0
Oct 30 20:18:18.042212 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:18.042185    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf" event={"ID":"ad047721-5c69-4d66-b297-6afe97b8b803","Type":"ContainerDied","Data":"d56b8886447abc9217eb497cd78afebddf3d91e170f6b6536c1b87ea83c8e03e"}
Oct 30 20:18:18.065115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:18.062367    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs" podStartSLOduration=2.062351933 podStartE2EDuration="2.062351933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:18.06092565 +0000 UTC m=+830.964345676" watchObservedRunningTime="2026-10-30 20:18:18.062351933 +0000 UTC m=+830.965771959"
Oct 30 20:18:18.749731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:18.749686    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:18:19.049194 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.049160    6058 generic.go:356] "Generic (PLEG): container finished" podID="582fd540-84d2-4fe7-aa89-e8e5212710c8" containerID="4cdce80d94d7b2141f71f36c158e9089fd6c97e09a76f0750d1961229647a853" exitCode=0
Oct 30 20:18:19.049667 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.049244    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs" event={"ID":"582fd540-84d2-4fe7-aa89-e8e5212710c8","Type":"ContainerDied","Data":"4cdce80d94d7b2141f71f36c158e9089fd6c97e09a76f0750d1961229647a853"}
Oct 30 20:18:19.196410 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.196381    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:19.270415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.270369    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca\") pod \"ad047721-5c69-4d66-b297-6afe97b8b803\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") "
Oct 30 20:18:19.270596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.270426    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9\" (UniqueName: \"kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9\") pod \"ad047721-5c69-4d66-b297-6afe97b8b803\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") "
Oct 30 20:18:19.270596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.270550    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca\") pod \"ad047721-5c69-4d66-b297-6afe97b8b803\" (UID: \"ad047721-5c69-4d66-b297-6afe97b8b803\") "
Oct 30 20:18:19.270919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.270891    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca" pod "ad047721-5c69-4d66-b297-6afe97b8b803" (UID: "ad047721-5c69-4d66-b297-6afe97b8b803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:19.271076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.271030    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:19.272559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.272530    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca" pod "ad047721-5c69-4d66-b297-6afe97b8b803" (UID: "ad047721-5c69-4d66-b297-6afe97b8b803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:19.272682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.272664    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9" pod "ad047721-5c69-4d66-b297-6afe97b8b803" (UID: "ad047721-5c69-4d66-b297-6afe97b8b803"). InnerVolumeSpecName "kube-api-access-ptvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:19.372400 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.372357    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad047721-5c69-4d66-b297-6afe97b8b803-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:19.372557 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:19.372402    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvj9\" (UniqueName: \"kubernetes.io/projected/ad047721-5c69-4d66-b297-6afe97b8b803-kube-api-access-ptvj9\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:19.655203 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:18:19.655114    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:18:20.088633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.088539    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"
Oct 30 20:18:20.107242 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.103083    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf" event={"ID":"ad047721-5c69-4d66-b297-6afe97b8b803","Type":"ContainerDied","Data":"b243a6be59ee91b28717a8fd55c6d3c2d5f8ab91037504f01809c7fb6a1d41bc"}
Oct 30 20:18:20.107242 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.107085    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b243a6be59ee91b28717a8fd55c6d3c2d5f8ab91037504f01809c7fb6a1d41bc"
Oct 30 20:18:20.641213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.641181    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:20.786411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.786372    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca\") pod \"582fd540-84d2-4fe7-aa89-e8e5212710c8\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") "
Oct 30 20:18:20.803643 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.803570    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl\" (UniqueName: \"kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl\") pod \"582fd540-84d2-4fe7-aa89-e8e5212710c8\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") "
Oct 30 20:18:20.803913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.803892    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca\") pod \"582fd540-84d2-4fe7-aa89-e8e5212710c8\" (UID: \"582fd540-84d2-4fe7-aa89-e8e5212710c8\") "
Oct 30 20:18:20.804887 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.803431    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca" pod "582fd540-84d2-4fe7-aa89-e8e5212710c8" (UID: "582fd540-84d2-4fe7-aa89-e8e5212710c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:20.908029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.905890    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:20.937387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.937187    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl" pod "582fd540-84d2-4fe7-aa89-e8e5212710c8" (UID: "582fd540-84d2-4fe7-aa89-e8e5212710c8"). InnerVolumeSpecName "kube-api-access-f8rfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:20.953287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:20.952513    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca" pod "582fd540-84d2-4fe7-aa89-e8e5212710c8" (UID: "582fd540-84d2-4fe7-aa89-e8e5212710c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:21.040929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:21.040807    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582fd540-84d2-4fe7-aa89-e8e5212710c8-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.041207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:21.041186    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8rfl\" (UniqueName: \"kubernetes.io/projected/582fd540-84d2-4fe7-aa89-e8e5212710c8-kube-api-access-f8rfl\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.127033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:21.126673    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs" event={"ID":"582fd540-84d2-4fe7-aa89-e8e5212710c8","Type":"ContainerDied","Data":"14bc44b9401f84cc9e85fd873e5cf076a2152bb3cd94c9b25f14a99be1f7e181"}
Oct 30 20:18:21.127033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:21.126734    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14bc44b9401f84cc9e85fd873e5cf076a2152bb3cd94c9b25f14a99be1f7e181"
Oct 30 20:18:21.127033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:21.126806    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"
Oct 30 20:18:24.233725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.233692    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"]
Oct 30 20:18:24.234458 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.234436    6058 cpu_manager.go:414] "containerMap: removing container" podUID="ad047721-5c69-4d66-b297-6afe97b8b803" containerName="revoke-keys"
Oct 30 20:18:24.234458 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.234457    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad047721-5c69-4d66-b297-6afe97b8b803" containerName="revoke-keys"
Oct 30 20:18:24.234637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.234504    6058 cpu_manager.go:414] "containerMap: removing container" podUID="582fd540-84d2-4fe7-aa89-e8e5212710c8" containerName="revoke-keys"
Oct 30 20:18:24.234637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.234513    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582fd540-84d2-4fe7-aa89-e8e5212710c8" containerName="revoke-keys"
Oct 30 20:18:24.234637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.234627    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="582fd540-84d2-4fe7-aa89-e8e5212710c8" containerName="revoke-keys"
Oct 30 20:18:24.234757 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.234650    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad047721-5c69-4d66-b297-6afe97b8b803" containerName="revoke-keys"
Oct 30 20:18:24.239134 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.239113    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262050    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262248 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262124    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262248 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262201    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcdk\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262316    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262363    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262399    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262428    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262469    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.262634 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.262541    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.273508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.273478    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"]
Oct 30 20:18:24.363628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363598    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.363812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363639    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.363812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363755    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.363812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363808    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.363980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363873    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.363980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363924    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.363980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363956    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.363977    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcdk\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.364055    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.364170    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.364217    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.364235    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.364332    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.364679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.364657    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.366394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.366358    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.366522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.366463    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.383144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.383114    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.383238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.383220    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcdk\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.552846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.552764    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:24.845718 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.845679    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"]
Oct 30 20:18:24.846339 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:24.846312    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbee7645_28fd_4fea_b29a_3956a82e7a8b.slice/crio-bbf4720849fd9a6d42808dc039bd9499ff574e832bf992a8c8b660c1535231a4 WatchSource:0}: Error finding container bbf4720849fd9a6d42808dc039bd9499ff574e832bf992a8c8b660c1535231a4: Status 404 returned error can't find the container with id bbf4720849fd9a6d42808dc039bd9499ff574e832bf992a8c8b660c1535231a4
Oct 30 20:18:24.848685 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.848652    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:24.848800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.848730    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:24.848800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:24.848774    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:25.174549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:25.174511    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" event={"ID":"dbee7645-28fd-4fea-b29a-3956a82e7a8b","Type":"ContainerStarted","Data":"573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303"}
Oct 30 20:18:25.176093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:25.176055    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" event={"ID":"dbee7645-28fd-4fea-b29a-3956a82e7a8b","Type":"ContainerStarted","Data":"bbf4720849fd9a6d42808dc039bd9499ff574e832bf992a8c8b660c1535231a4"}
Oct 30 20:18:25.199683 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:25.199618    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podStartSLOduration=1.199597604 podStartE2EDuration="1.199597604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:25.199176956 +0000 UTC m=+838.102596993" watchObservedRunningTime="2026-10-30 20:18:25.199597604 +0000 UTC m=+838.103017630"
Oct 30 20:18:25.555089 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:25.553298    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:25.558626 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:25.558288    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:18:25.558626 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:25.558350    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 20:18:26.560029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:26.555307    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:18:26.560029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:26.555379    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 20:18:27.080697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.080657    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"]
Oct 30 20:18:27.085118 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.085092    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.091469 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.091192    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:27.091626 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.091596    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"]
Oct 30 20:18:27.092207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.092189    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:27.092311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.092268    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:27.092362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.092342    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:18:27.098345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.095403    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.098345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.095451    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.098345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.095513    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.199260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.199217    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.199440 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.199332    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.199510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.199479    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.202279 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.202254    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:27.202279 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.202257    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:27.210492 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.210466    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.211805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.211780    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.213323 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.213304    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:27.222899 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.222880    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg\") pod \"maas-api-revoke-sub-1600ffe8-xxwxs\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.415213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.415184    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:18:27.424946 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.424919    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:27.589030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.585759    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:18:27.589030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.585834    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 20:18:27.734887 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:27.734860    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"]
Oct 30 20:18:27.736308 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:27.736282    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ec849f_8646_4935_ba77_737a26790c78.slice/crio-4d3e6ab1102eb481be4034d2f76b487edf1e59ba9970eadc42bfcb0aabc2ae13 WatchSource:0}: Error finding container 4d3e6ab1102eb481be4034d2f76b487edf1e59ba9970eadc42bfcb0aabc2ae13: Status 404 returned error can't find the container with id 4d3e6ab1102eb481be4034d2f76b487edf1e59ba9970eadc42bfcb0aabc2ae13
Oct 30 20:18:28.200807 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:28.200773    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs" event={"ID":"15ec849f-8646-4935-ba77-737a26790c78","Type":"ContainerStarted","Data":"2245458b4d271fbeafb6d93155269d24b31da0a567983a45b6cb07a90102cc5d"}
Oct 30 20:18:28.200912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:28.200837    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs" event={"ID":"15ec849f-8646-4935-ba77-737a26790c78","Type":"ContainerStarted","Data":"4d3e6ab1102eb481be4034d2f76b487edf1e59ba9970eadc42bfcb0aabc2ae13"}
Oct 30 20:18:28.223602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:28.223558    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs" podStartSLOduration=1.223544318 podStartE2EDuration="1.223544318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:28.221687844 +0000 UTC m=+841.125107863" watchObservedRunningTime="2026-10-30 20:18:28.223544318 +0000 UTC m=+841.126964367"
Oct 30 20:18:28.557090 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:28.555324    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:18:28.557090 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:28.555393    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 20:18:29.209858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:29.209819    6058 generic.go:356] "Generic (PLEG): container finished" podID="15ec849f-8646-4935-ba77-737a26790c78" containerID="2245458b4d271fbeafb6d93155269d24b31da0a567983a45b6cb07a90102cc5d" exitCode=0
Oct 30 20:18:29.212413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:29.212383    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs" event={"ID":"15ec849f-8646-4935-ba77-737a26790c78","Type":"ContainerDied","Data":"2245458b4d271fbeafb6d93155269d24b31da0a567983a45b6cb07a90102cc5d"}
Oct 30 20:18:29.557841 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:29.557358    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:18:29.557841 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:29.557420    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 20:18:30.415907 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.415881    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:30.441308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.435724    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg\") pod \"15ec849f-8646-4935-ba77-737a26790c78\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") "
Oct 30 20:18:30.441308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.435788    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca\") pod \"15ec849f-8646-4935-ba77-737a26790c78\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") "
Oct 30 20:18:30.441308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.435827    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca\") pod \"15ec849f-8646-4935-ba77-737a26790c78\" (UID: \"15ec849f-8646-4935-ba77-737a26790c78\") "
Oct 30 20:18:30.441308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.436715    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca" pod "15ec849f-8646-4935-ba77-737a26790c78" (UID: "15ec849f-8646-4935-ba77-737a26790c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:30.441308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.438775    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg" pod "15ec849f-8646-4935-ba77-737a26790c78" (UID: "15ec849f-8646-4935-ba77-737a26790c78"). InnerVolumeSpecName "kube-api-access-2xwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:30.441308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.440125    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca" pod "15ec849f-8646-4935-ba77-737a26790c78" (UID: "15ec849f-8646-4935-ba77-737a26790c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:30.537499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.537423    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwmg\" (UniqueName: \"kubernetes.io/projected/15ec849f-8646-4935-ba77-737a26790c78-kube-api-access-2xwmg\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:30.537499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.537451    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ec849f-8646-4935-ba77-737a26790c78-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:30.537499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.537461    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ec849f-8646-4935-ba77-737a26790c78-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:30.579144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.579110    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:18:30.579418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:30.579390    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 20:18:31.227975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:31.227935    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"
Oct 30 20:18:31.227975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:31.227959    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs" event={"ID":"15ec849f-8646-4935-ba77-737a26790c78","Type":"ContainerDied","Data":"4d3e6ab1102eb481be4034d2f76b487edf1e59ba9970eadc42bfcb0aabc2ae13"}
Oct 30 20:18:31.228169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:31.228032    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d3e6ab1102eb481be4034d2f76b487edf1e59ba9970eadc42bfcb0aabc2ae13"
Oct 30 20:18:31.560672 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:31.560584    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:31.561426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:31.561298    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:31.562733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:31.562712    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:18:34.497454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:34.497403    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:18:39.917990 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.917949    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"]
Oct 30 20:18:39.918623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.918600    6058 cpu_manager.go:414] "containerMap: removing container" podUID="15ec849f-8646-4935-ba77-737a26790c78" containerName="revoke-keys"
Oct 30 20:18:39.918722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.918625    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ec849f-8646-4935-ba77-737a26790c78" containerName="revoke-keys"
Oct 30 20:18:39.918814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.918801    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ec849f-8646-4935-ba77-737a26790c78" containerName="revoke-keys"
Oct 30 20:18:39.923306 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.923283    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:39.929275 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.929253    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-44135b-openshift-default-dockercfg-sp9bd\""
Oct 30 20:18:39.935677 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:39.935653    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"]
Oct 30 20:18:40.078710 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078676    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.078858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078729    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27js9\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.078858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078753    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.078858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078832    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.078980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078878    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.078980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078929    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.078980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.078958    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.079114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.079031    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.079114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.079071    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.179793 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.179708    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.179793 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.179762    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.179977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.179938    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180059 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180032    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27js9\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180081    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180116    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180156    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180210    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180265    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180426    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180723 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180698    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180723 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180716    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180753    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.180958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.180934    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.182081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.182057    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.182332 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.182310    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.194326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.194304    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27js9\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.194430 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.194327    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.237984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.237959    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:40.531095 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.531063    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"]
Oct 30 20:18:40.532436 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:40.532409    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cef02b6_4044_45dc_b462_d829bf44778b.slice/crio-e023c33df9a64ea3c9d274ae32a01fd237baf2bee7882f502cb13df96758081b WatchSource:0}: Error finding container e023c33df9a64ea3c9d274ae32a01fd237baf2bee7882f502cb13df96758081b: Status 404 returned error can't find the container with id e023c33df9a64ea3c9d274ae32a01fd237baf2bee7882f502cb13df96758081b
Oct 30 20:18:40.536588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.535577    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:40.536588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.535744    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:40.536588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:40.535834    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:41.315347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.315312    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"]
Oct 30 20:18:41.319163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.319143    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.336361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.328249    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:41.336361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.328661    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"]
Oct 30 20:18:41.336361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.329254    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:41.336361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.329389    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:18:41.336361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.329442    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:41.388895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.388859    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" event={"ID":"9cef02b6-4044-45dc-b462-d829bf44778b","Type":"ContainerStarted","Data":"86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136"}
Oct 30 20:18:41.389126 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.388921    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" event={"ID":"9cef02b6-4044-45dc-b462-d829bf44778b","Type":"ContainerStarted","Data":"e023c33df9a64ea3c9d274ae32a01fd237baf2bee7882f502cb13df96758081b"}
Oct 30 20:18:41.414204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.414151    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podStartSLOduration=2.414136686 podStartE2EDuration="2.414136686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:41.412726255 +0000 UTC m=+854.316146282" watchObservedRunningTime="2026-10-30 20:18:41.414136686 +0000 UTC m=+854.317556751"
Oct 30 20:18:41.496490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.496443    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjqz\" (UniqueName: \"kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.496666 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.496550    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.496666 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.496599    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.598442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.598348    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjqz\" (UniqueName: \"kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.598442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.598438    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.598676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.598480    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.599320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.599292    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.600975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.600954    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.613488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.613462    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjqz\" (UniqueName: \"kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz\") pod \"maas-api-revoke-sub-b87cd34e-2n5dk\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.635066 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.635045    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:41.970417 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:41.970366    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fdeb58_8c26_4e30_abdd_1ae63affb174.slice/crio-28ab3334a5ff9c8e2407ac303780b1c3c7ace9d51abde1f33d85ade1fcb49498 WatchSource:0}: Error finding container 28ab3334a5ff9c8e2407ac303780b1c3c7ace9d51abde1f33d85ade1fcb49498: Status 404 returned error can't find the container with id 28ab3334a5ff9c8e2407ac303780b1c3c7ace9d51abde1f33d85ade1fcb49498
Oct 30 20:18:41.985661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:41.985630    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"]
Oct 30 20:18:42.238617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:42.238524    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:42.240351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:42.240326    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:18:42.240493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:42.240370    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 20:18:42.397687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:42.397648    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk" event={"ID":"b9fdeb58-8c26-4e30-abdd-1ae63affb174","Type":"ContainerStarted","Data":"0a24c0cf959b18c99fb044083432d5ffa8b8811736058cd7856a8b92b6f5a8ac"}
Oct 30 20:18:42.398075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:42.397718    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk" event={"ID":"b9fdeb58-8c26-4e30-abdd-1ae63affb174","Type":"ContainerStarted","Data":"28ab3334a5ff9c8e2407ac303780b1c3c7ace9d51abde1f33d85ade1fcb49498"}
Oct 30 20:18:42.418017 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:42.417955    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk" podStartSLOduration=1.417941726 podStartE2EDuration="1.417941726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:42.415901692 +0000 UTC m=+855.319321719" watchObservedRunningTime="2026-10-30 20:18:42.417941726 +0000 UTC m=+855.321361751"
Oct 30 20:18:43.238646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:43.238614    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:18:43.238789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:43.238666    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 20:18:43.407472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:43.407436    6058 generic.go:356] "Generic (PLEG): container finished" podID="b9fdeb58-8c26-4e30-abdd-1ae63affb174" containerID="0a24c0cf959b18c99fb044083432d5ffa8b8811736058cd7856a8b92b6f5a8ac" exitCode=0
Oct 30 20:18:43.407844 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:43.407519    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk" event={"ID":"b9fdeb58-8c26-4e30-abdd-1ae63affb174","Type":"ContainerDied","Data":"0a24c0cf959b18c99fb044083432d5ffa8b8811736058cd7856a8b92b6f5a8ac"}
Oct 30 20:18:44.238565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.238526    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:18:44.238718 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.238582    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 20:18:44.610400 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.610377    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:44.737343 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.737313    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz\" (UniqueName: \"kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz\") pod \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") "
Oct 30 20:18:44.737494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.737374    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca\") pod \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") "
Oct 30 20:18:44.737494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.737415    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca\") pod \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\" (UID: \"b9fdeb58-8c26-4e30-abdd-1ae63affb174\") "
Oct 30 20:18:44.737841 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.737815    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca" pod "b9fdeb58-8c26-4e30-abdd-1ae63affb174" (UID: "b9fdeb58-8c26-4e30-abdd-1ae63affb174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:44.739476 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.739452    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz" pod "b9fdeb58-8c26-4e30-abdd-1ae63affb174" (UID: "b9fdeb58-8c26-4e30-abdd-1ae63affb174"). InnerVolumeSpecName "kube-api-access-zrjqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:44.739573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.739527    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca" pod "b9fdeb58-8c26-4e30-abdd-1ae63affb174" (UID: "b9fdeb58-8c26-4e30-abdd-1ae63affb174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:44.838541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.838501    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjqz\" (UniqueName: \"kubernetes.io/projected/b9fdeb58-8c26-4e30-abdd-1ae63affb174-kube-api-access-zrjqz\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:44.838541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.838529    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:44.838541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:44.838539    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fdeb58-8c26-4e30-abdd-1ae63affb174-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:45.238642 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:45.238611    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:18:45.238767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:45.238659    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 20:18:45.421141 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:45.421108    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"
Oct 30 20:18:45.421285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:45.421139    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk" event={"ID":"b9fdeb58-8c26-4e30-abdd-1ae63affb174","Type":"ContainerDied","Data":"28ab3334a5ff9c8e2407ac303780b1c3c7ace9d51abde1f33d85ade1fcb49498"}
Oct 30 20:18:45.421285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:45.421189    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ab3334a5ff9c8e2407ac303780b1c3c7ace9d51abde1f33d85ade1fcb49498"
Oct 30 20:18:46.239169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:46.239134    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:18:46.239647 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:46.239187    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 20:18:46.863030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:46.862268    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:18:47.759114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:47.759064    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:47.761304 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:47.761284    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:47.762495 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:47.762474    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:18:49.660052 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:18:49.659986    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:18:54.615880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.615843    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"]
Oct 30 20:18:54.616496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.616479    6058 cpu_manager.go:414] "containerMap: removing container" podUID="b9fdeb58-8c26-4e30-abdd-1ae63affb174" containerName="revoke-keys"
Oct 30 20:18:54.616577 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.616497    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fdeb58-8c26-4e30-abdd-1ae63affb174" containerName="revoke-keys"
Oct 30 20:18:54.616639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.616586    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fdeb58-8c26-4e30-abdd-1ae63affb174" containerName="revoke-keys"
Oct 30 20:18:54.621676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.621654    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.638826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.638802    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"]
Oct 30 20:18:54.668196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668161    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668205    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668348    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668390    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668477    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668517    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668686 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668670    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668707    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.668833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.668810    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.740795 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.740765    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"]
Oct 30 20:18:54.745196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.745170    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.759564 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.759488    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"]
Oct 30 20:18:54.791566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.791527    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.791813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.791794    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.791941 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.791927    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.792091 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792074    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.792217 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792202    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.792462 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792441    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.792601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792586    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.792702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792688    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvnxb\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.792817 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792802    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.792909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.792895    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.793056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793041    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.793163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793150    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.793305 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793290    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.793416 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793402    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.793514 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793501    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.793625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793613    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.793726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793712    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.793853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.793840    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.794450 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.794427    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.794815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.794793    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.795627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.795606    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.797558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.796731    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.797558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.796966    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.805487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.805467    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.811349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.807505    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.811349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.807854    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.811349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.809620    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.895522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895437    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895476    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895507    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895558    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895631    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895693    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895732    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.895770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895759    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.896108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.895782    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvnxb\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.896108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.896042    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.896346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.896319    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.896416 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.896320    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.896416 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.896360    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.896732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.896708    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.898222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.898196    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.898568 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.898544    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.912016 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.911979    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvnxb\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.914871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.914850    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token\") pod \"e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") " pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:54.935837 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.935816    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:54.958604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.958062    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"]
Oct 30 20:18:54.963138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.963116    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:54.977763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:54.974572    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"]
Oct 30 20:18:55.061279 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.061236    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:55.101368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101338    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kjp\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101391    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101528    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101606    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101712 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101654    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101822    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101921    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.101987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.101971    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.102057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.102042    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.202998    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203105    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203127    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203217    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203295    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203343    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203385    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203440    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kjp\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203492    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203634    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203758 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203731    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203886    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.203913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203898    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.204123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.203928    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.205713 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.205690    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.205838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.205697    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.215997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.215973    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.216188 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.216169    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kjp\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.234584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.234563    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"]
Oct 30 20:18:55.236093 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:55.236069    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51195988_aabb_45b5_be2c_332ad185258c.slice/crio-b4aed418d994b0eb1bc541c137fd230f1e198df44e4d8f2ab083703100d666a2 WatchSource:0}: Error finding container b4aed418d994b0eb1bc541c137fd230f1e198df44e4d8f2ab083703100d666a2: Status 404 returned error can't find the container with id b4aed418d994b0eb1bc541c137fd230f1e198df44e4d8f2ab083703100d666a2
Oct 30 20:18:55.238397 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.238366    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.238481 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.238429    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.238520 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.238478    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.306847 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.306824    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:55.351630 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.351521    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"]
Oct 30 20:18:55.360428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.353315    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.360428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.353392    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.360428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.353434    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.498822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.498404    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" event={"ID":"a869dc70-8829-44c3-9810-6045910e233b","Type":"ContainerStarted","Data":"b6db5f2729ccc9c59140ccd4eb4f26c5166da06bdafb8e48476ec7c1aa79fcc8"}
Oct 30 20:18:55.498822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.498468    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" event={"ID":"a869dc70-8829-44c3-9810-6045910e233b","Type":"ContainerStarted","Data":"4862093f82b6db01b08992f8aeb4f953693ed2b06412c9364c8d434e409bac06"}
Oct 30 20:18:55.502044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.501881    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" event={"ID":"51195988-aabb-45b5-be2c-332ad185258c","Type":"ContainerStarted","Data":"864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19"}
Oct 30 20:18:55.502044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.501935    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" event={"ID":"51195988-aabb-45b5-be2c-332ad185258c","Type":"ContainerStarted","Data":"b4aed418d994b0eb1bc541c137fd230f1e198df44e4d8f2ab083703100d666a2"}
Oct 30 20:18:55.528679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.525232    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podStartSLOduration=1.525218032 podStartE2EDuration="1.525218032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:55.524607301 +0000 UTC m=+868.428027326" watchObservedRunningTime="2026-10-30 20:18:55.525218032 +0000 UTC m=+868.428638059"
Oct 30 20:18:55.559109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.558993    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podStartSLOduration=1.558975301 podStartE2EDuration="1.558975301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:55.554800765 +0000 UTC m=+868.458220797" watchObservedRunningTime="2026-10-30 20:18:55.558975301 +0000 UTC m=+868.462395332"
Oct 30 20:18:55.609719 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.609693    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"]
Oct 30 20:18:55.613000 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:18:55.612969    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97445ce_898f_47b9_84d1_df8ee2215258.slice/crio-eae718cf000ce5419de3e97d78d09f4306e3310e8ec2ddee7f1b8afcba9aea7d WatchSource:0}: Error finding container eae718cf000ce5419de3e97d78d09f4306e3310e8ec2ddee7f1b8afcba9aea7d: Status 404 returned error can't find the container with id eae718cf000ce5419de3e97d78d09f4306e3310e8ec2ddee7f1b8afcba9aea7d
Oct 30 20:18:55.615224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.615187    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.615403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.615378    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.615552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.615532    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:18:55.936801 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:55.936769    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:18:56.061654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.061625    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:18:56.062847 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.062821    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:18:56.062959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.062872    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:18:56.546106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.546057    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" event={"ID":"f97445ce-898f-47b9-84d1-df8ee2215258","Type":"ContainerStarted","Data":"94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737"}
Oct 30 20:18:56.546280 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.546123    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" event={"ID":"f97445ce-898f-47b9-84d1-df8ee2215258","Type":"ContainerStarted","Data":"eae718cf000ce5419de3e97d78d09f4306e3310e8ec2ddee7f1b8afcba9aea7d"}
Oct 30 20:18:56.944102 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.938829    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:18:56.944102 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:56.938923    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:18:57.077029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.070625    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:18:57.077029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.070685    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:18:57.309262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.309149    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:18:57.309608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.309572    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:18:57.309774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.309748    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:18:57.936436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.936399    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:18:57.936730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:57.936695    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:18:58.063434 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:58.063396    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:18:58.064352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:58.064306    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:18:58.308083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:58.307421    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:18:58.308083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:58.307484    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:18:58.938405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:58.938365    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:18:58.938725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:58.938695    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:18:59.066029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:59.063278    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:18:59.066029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:59.063342    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:18:59.327085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:59.308489    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:18:59.327085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:59.309191    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:18:59.946408 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:59.946369    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:18:59.946728 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:18:59.946690    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:19:00.072115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.071616    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:19:00.072115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.071712    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:19:00.307828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.307334    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:19:00.307828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.307397    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:19:00.452117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.451089    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:00.556143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.556080    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:00.602202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.601122    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:00.663990 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.663142    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:00.693160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.693104    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:00.735037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.711439    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:00.940035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.939546    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:19:00.940035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:00.939639    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:19:01.071031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.061659    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:19:01.071031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.061717    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:19:01.307420 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.307332    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:19:01.307955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.307924    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:19:01.458133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.458090    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:19:01.459560 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.459536    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:19:01.580947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.489260    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podStartSLOduration=7.489240369 podStartE2EDuration="7.489240369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:56.588377144 +0000 UTC m=+869.491797170" watchObservedRunningTime="2026-10-30 20:19:01.489240369 +0000 UTC m=+874.392660396"
Oct 30 20:19:01.698160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.698113    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:01.722666 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.699676    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:01.727682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.727652    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:19:01.727916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.727900    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:19:01.728371 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.699713    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:19:01.729193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.699736    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:19:01.864084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.701753    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:19:01.864084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.721695    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:01.864084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.749915    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:19:01.864084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.846516    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:19:01.864084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:01.846676    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:19:02.199093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.198320    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:19:02.199093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.198423    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:19:02.199093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.198515    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:19:02.199093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.198550    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:19:02.199093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.198634    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded"
Oct 30 20:19:02.446216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.446180    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:19:02.446785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.446756    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:19:02.534083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.527630    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:19:02.746029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.742128    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:02.782028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.774522    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:02.782028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.775717    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:19:02.866088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.850460    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:02.866088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.850837    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:19:02.866088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.851807    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:19:02.866088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.852152    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:19:02.955029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.954787    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:19:02.955029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:02.954868    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:19:03.032780 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.032743    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"]
Oct 30 20:19:03.039698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.039670    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"]
Oct 30 20:19:03.039968 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.039952    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.096413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.085193    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:19:03.096616 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.096501    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:19:03.111307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.110396    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:19:03.111880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.111585    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhhd\" (UniqueName: \"kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.111880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.111697    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.111880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.111759    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.133077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.132334    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:19:03.134742 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.134714    6058 patch_prober.go:29] interesting pod/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:19:03.134957 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.134929    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:19:03.156033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.147535    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:03.160149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.160123    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:03.160748 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.160575    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:03.160932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.160672    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:03.161074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.160887    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:19:03.192629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.192595    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:19:03.213658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.212784    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.213658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.212983    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhhd\" (UniqueName: \"kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.213658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.213106    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.214869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.214841    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.259174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.259132    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.265670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.265642    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhhd\" (UniqueName: \"kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd\") pod \"maas-api-revoke-sub-43b937f7-gmk46\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.280037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.279974    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:19:03.315378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.315344    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:19:03.315699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.315670    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 20:19:03.439255 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.439174    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:03.812028 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:03.811972    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0f2407_8e02_464f_84cc_72e94fc1244e.slice/crio-6bc02f8e61b2071edf8165c846359b17e332f7f52ee6f8374a92edf4710ec66b WatchSource:0}: Error finding container 6bc02f8e61b2071edf8165c846359b17e332f7f52ee6f8374a92edf4710ec66b: Status 404 returned error can't find the container with id 6bc02f8e61b2071edf8165c846359b17e332f7f52ee6f8374a92edf4710ec66b
Oct 30 20:19:03.812450 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.812280    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"]
Oct 30 20:19:03.937274 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.937225    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:19:03.937477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:03.937292    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 20:19:04.505755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.505702    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46" event={"ID":"5b0f2407-8e02-464f-84cc-72e94fc1244e","Type":"ContainerStarted","Data":"1dd0b2bea4d82a537dfd84106efeeb45f238062ba387152d607b1e4aa81c0a19"}
Oct 30 20:19:04.506080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.506056    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46" event={"ID":"5b0f2407-8e02-464f-84cc-72e94fc1244e","Type":"ContainerStarted","Data":"6bc02f8e61b2071edf8165c846359b17e332f7f52ee6f8374a92edf4710ec66b"}
Oct 30 20:19:04.532466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.531163    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:19:04.532466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.532106    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:19:04.535576 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.535545    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:19:04.537927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.537881    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46" podStartSLOduration=2.537868912 podStartE2EDuration="2.537868912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:04.535775684 +0000 UTC m=+877.439195711" watchObservedRunningTime="2026-10-30 20:19:04.537868912 +0000 UTC m=+877.441288937"
Oct 30 20:19:04.747930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.747891    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:19:04.748318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.748292    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:19:04.749554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.749530    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:19:04.946526 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.946491    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:19:04.947581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.947558    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:19:04.948986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:04.948967    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:19:05.515900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:05.515861    6058 generic.go:356] "Generic (PLEG): container finished" podID="5b0f2407-8e02-464f-84cc-72e94fc1244e" containerID="1dd0b2bea4d82a537dfd84106efeeb45f238062ba387152d607b1e4aa81c0a19" exitCode=0
Oct 30 20:19:05.516103 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:05.515949    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46" event={"ID":"5b0f2407-8e02-464f-84cc-72e94fc1244e","Type":"ContainerDied","Data":"1dd0b2bea4d82a537dfd84106efeeb45f238062ba387152d607b1e4aa81c0a19"}
Oct 30 20:19:06.669463 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.669438    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:06.805701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.805626    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca\") pod \"5b0f2407-8e02-464f-84cc-72e94fc1244e\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") "
Oct 30 20:19:06.805701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.805684    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca\") pod \"5b0f2407-8e02-464f-84cc-72e94fc1244e\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") "
Oct 30 20:19:06.805898 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.805745    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd\" (UniqueName: \"kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd\") pod \"5b0f2407-8e02-464f-84cc-72e94fc1244e\" (UID: \"5b0f2407-8e02-464f-84cc-72e94fc1244e\") "
Oct 30 20:19:06.806058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.806035    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca" pod "5b0f2407-8e02-464f-84cc-72e94fc1244e" (UID: "5b0f2407-8e02-464f-84cc-72e94fc1244e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:06.806340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.806315    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:06.807861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.807833    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca" pod "5b0f2407-8e02-464f-84cc-72e94fc1244e" (UID: "5b0f2407-8e02-464f-84cc-72e94fc1244e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:06.807956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.807863    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd" pod "5b0f2407-8e02-464f-84cc-72e94fc1244e" (UID: "5b0f2407-8e02-464f-84cc-72e94fc1244e"). InnerVolumeSpecName "kube-api-access-fzhhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:06.907621 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.907591    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0f2407-8e02-464f-84cc-72e94fc1244e-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:06.907621 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:06.907618    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhhd\" (UniqueName: \"kubernetes.io/projected/5b0f2407-8e02-464f-84cc-72e94fc1244e-kube-api-access-fzhhd\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:07.530892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:07.530812    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46" event={"ID":"5b0f2407-8e02-464f-84cc-72e94fc1244e","Type":"ContainerDied","Data":"6bc02f8e61b2071edf8165c846359b17e332f7f52ee6f8374a92edf4710ec66b"}
Oct 30 20:19:07.530892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:07.530867    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc02f8e61b2071edf8165c846359b17e332f7f52ee6f8374a92edf4710ec66b"
Oct 30 20:19:07.530892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:07.530834    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"
Oct 30 20:19:08.253079 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.253032    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"]
Oct 30 20:19:08.253564 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.253551    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5b0f2407-8e02-464f-84cc-72e94fc1244e" containerName="revoke-keys"
Oct 30 20:19:08.253604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.253566    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0f2407-8e02-464f-84cc-72e94fc1244e" containerName="revoke-keys"
Oct 30 20:19:08.253676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.253667    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0f2407-8e02-464f-84cc-72e94fc1244e" containerName="revoke-keys"
Oct 30 20:19:08.257177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.257162    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.263600 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.263568    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:08.263757 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.263642    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.263757 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.263736    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.263883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.263841    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:08.266224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.266202    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"]
Oct 30 20:19:08.325167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.325137    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.325594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.325572    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.325734 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.325717    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gldh\" (UniqueName: \"kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.427341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.427310    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.427472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.427359    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gldh\" (UniqueName: \"kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.427472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.427394    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.428117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.428090    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.429671 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.429648    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.440348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.440325    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gldh\" (UniqueName: \"kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh\") pod \"maas-api-revoke-sub-a626424d-jbt29\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.573945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.573857    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:08.854150 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:08.854117    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"]
Oct 30 20:19:08.854271 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:08.854196    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7cf03a_b513_44f7_b335_b80725980d6a.slice/crio-c06cf6e61eb3e8929cd02e06ffaa912558b0eb747840d085fc9bbb75bf99b0e4 WatchSource:0}: Error finding container c06cf6e61eb3e8929cd02e06ffaa912558b0eb747840d085fc9bbb75bf99b0e4: Status 404 returned error can't find the container with id c06cf6e61eb3e8929cd02e06ffaa912558b0eb747840d085fc9bbb75bf99b0e4
Oct 30 20:19:09.548348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:09.548246    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29" event={"ID":"1f7cf03a-b513-44f7-b335-b80725980d6a","Type":"ContainerStarted","Data":"6d386e0eef22d8304b90848c172a5a07d4cb88d55bfc1390164dacebf5f0364c"}
Oct 30 20:19:09.548348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:09.548316    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29" event={"ID":"1f7cf03a-b513-44f7-b335-b80725980d6a","Type":"ContainerStarted","Data":"c06cf6e61eb3e8929cd02e06ffaa912558b0eb747840d085fc9bbb75bf99b0e4"}
Oct 30 20:19:09.579426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:09.579352    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29" podStartSLOduration=1.57933162 podStartE2EDuration="1.57933162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:09.575756243 +0000 UTC m=+882.479176284" watchObservedRunningTime="2026-10-30 20:19:09.57933162 +0000 UTC m=+882.482751646"
Oct 30 20:19:10.054618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.054585    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"]
Oct 30 20:19:10.058707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.058690    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.074277 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.074248    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"]
Oct 30 20:19:10.149944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.149909    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n54r\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.149970    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150099    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150123    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150158    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150292    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150337    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150455 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150427    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.150490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.150457    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252110 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252074    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n54r\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252156    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252205    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252239    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252293    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252345    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252380    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252580 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252429    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252580 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252455    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252692    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252802 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252782    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.252928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.252902    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.253085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.253031    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.253190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.253172    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.254699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.254674    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.254908 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.254891    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.276224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.276200    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n54r\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.276359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.276338    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.375112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.375083    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:10.556336 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.556299    6058 generic.go:356] "Generic (PLEG): container finished" podID="1f7cf03a-b513-44f7-b335-b80725980d6a" containerID="6d386e0eef22d8304b90848c172a5a07d4cb88d55bfc1390164dacebf5f0364c" exitCode=0
Oct 30 20:19:10.556813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.556349    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29" event={"ID":"1f7cf03a-b513-44f7-b335-b80725980d6a","Type":"ContainerDied","Data":"6d386e0eef22d8304b90848c172a5a07d4cb88d55bfc1390164dacebf5f0364c"}
Oct 30 20:19:10.671618 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:10.671587    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b92260e_31ee_4730_b5aa_88fcedcd6ea0.slice/crio-01e2387b760da938f0a36127b7d1e9322fcadbe13a598514d59c4dd4ecf3a1a5 WatchSource:0}: Error finding container 01e2387b760da938f0a36127b7d1e9322fcadbe13a598514d59c4dd4ecf3a1a5: Status 404 returned error can't find the container with id 01e2387b760da938f0a36127b7d1e9322fcadbe13a598514d59c4dd4ecf3a1a5
Oct 30 20:19:10.671717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.671683    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"]
Oct 30 20:19:10.674346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.674309    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:10.674424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.674391    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:10.674461 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:10.674433    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:11.588176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.588119    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" event={"ID":"2b92260e-31ee-4730-b5aa-88fcedcd6ea0","Type":"ContainerStarted","Data":"af07fa15878b19c2fa5ccdf805648d3336b81f98168adabc4f759a6e070faf98"}
Oct 30 20:19:11.588670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.588186    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" event={"ID":"2b92260e-31ee-4730-b5aa-88fcedcd6ea0","Type":"ContainerStarted","Data":"01e2387b760da938f0a36127b7d1e9322fcadbe13a598514d59c4dd4ecf3a1a5"}
Oct 30 20:19:11.618476 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.618417    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podStartSLOduration=1.618367594 podStartE2EDuration="1.618367594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:11.614668536 +0000 UTC m=+884.518088561" watchObservedRunningTime="2026-10-30 20:19:11.618367594 +0000 UTC m=+884.521787620"
Oct 30 20:19:11.802538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.802506    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:11.900391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.900352    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca\") pod \"1f7cf03a-b513-44f7-b335-b80725980d6a\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") "
Oct 30 20:19:11.900593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.900521    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh\" (UniqueName: \"kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh\") pod \"1f7cf03a-b513-44f7-b335-b80725980d6a\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") "
Oct 30 20:19:11.900655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.900619    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca\") pod \"1f7cf03a-b513-44f7-b335-b80725980d6a\" (UID: \"1f7cf03a-b513-44f7-b335-b80725980d6a\") "
Oct 30 20:19:11.900812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.900784    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca" pod "1f7cf03a-b513-44f7-b335-b80725980d6a" (UID: "1f7cf03a-b513-44f7-b335-b80725980d6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:11.901105 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.901086    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:11.902619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.902593    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca" pod "1f7cf03a-b513-44f7-b335-b80725980d6a" (UID: "1f7cf03a-b513-44f7-b335-b80725980d6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:11.902723 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:11.902678    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh" pod "1f7cf03a-b513-44f7-b335-b80725980d6a" (UID: "1f7cf03a-b513-44f7-b335-b80725980d6a"). InnerVolumeSpecName "kube-api-access-7gldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:12.002251 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.002214    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gldh\" (UniqueName: \"kubernetes.io/projected/1f7cf03a-b513-44f7-b335-b80725980d6a-kube-api-access-7gldh\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:12.002251 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.002247    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cf03a-b513-44f7-b335-b80725980d6a-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:12.378402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.378369    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:12.395794 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.395726    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:19:12.395968 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.395812    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 20:19:12.596605 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.596561    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"
Oct 30 20:19:12.596605 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.596592    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29" event={"ID":"1f7cf03a-b513-44f7-b335-b80725980d6a","Type":"ContainerDied","Data":"c06cf6e61eb3e8929cd02e06ffaa912558b0eb747840d085fc9bbb75bf99b0e4"}
Oct 30 20:19:12.597142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:12.596636    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c06cf6e61eb3e8929cd02e06ffaa912558b0eb747840d085fc9bbb75bf99b0e4"
Oct 30 20:19:13.383032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:13.382454    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:19:13.383032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:13.382515    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 20:19:14.376359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:14.376320    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:19:14.376927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:14.376896    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 20:19:15.376396 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:15.376358    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:19:15.377466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:15.377431    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 20:19:16.319511 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:16.319469    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:19:16.385074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:16.384158    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:19:16.385074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:16.384220    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 20:19:17.543937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:17.543908    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:17.545041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:17.544993    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:17.545814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:17.545785    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:19:19.657789 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:19:19.657752    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:19:19.736781 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.736747    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"]
Oct 30 20:19:19.737420 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.737400    6058 cpu_manager.go:414] "containerMap: removing container" podUID="1f7cf03a-b513-44f7-b335-b80725980d6a" containerName="revoke-keys"
Oct 30 20:19:19.737550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.737421    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7cf03a-b513-44f7-b335-b80725980d6a" containerName="revoke-keys"
Oct 30 20:19:19.737609 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.737557    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7cf03a-b513-44f7-b335-b80725980d6a" containerName="revoke-keys"
Oct 30 20:19:19.742747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.742721    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.749000 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.748975    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-75e734\"/\"distinct-2-w6-75e734-kserve-self-signed-certs\""
Oct 30 20:19:19.752903 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.752883    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"]
Oct 30 20:19:19.856776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.856741    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z924r\" (UniqueName: \"kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.856962 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.856795    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.857094 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.857066    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.857169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.857108    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.857226 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.857204    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.857293 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.857277    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.958925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.958840    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z924r\" (UniqueName: \"kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.958925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.958900    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959085    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959113    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959157    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959226    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959322    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959534 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959511    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.959602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.959562    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.961439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.961413    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.961758 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.961738    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:19.972662 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:19.972640    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z924r\" (UniqueName: \"kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r\") pod \"distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") " pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:20.062595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:20.062559    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:20.347287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:20.347254    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"]
Oct 30 20:19:20.349134 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:20.349107    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1731030_822c_4017_b1e5_62c7a8d013fa.slice/crio-53d6c4b0b9928d972167e54da458d43b321402cea8f6bde7e7b68c1ff2b921d5 WatchSource:0}: Error finding container 53d6c4b0b9928d972167e54da458d43b321402cea8f6bde7e7b68c1ff2b921d5: Status 404 returned error can't find the container with id 53d6c4b0b9928d972167e54da458d43b321402cea8f6bde7e7b68c1ff2b921d5
Oct 30 20:19:20.673799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:20.673755    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" event={"ID":"c1731030-822c-4017-b1e5-62c7a8d013fa","Type":"ContainerStarted","Data":"b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2"}
Oct 30 20:19:20.673799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:20.673814    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" event={"ID":"c1731030-822c-4017-b1e5-62c7a8d013fa","Type":"ContainerStarted","Data":"53d6c4b0b9928d972167e54da458d43b321402cea8f6bde7e7b68c1ff2b921d5"}
Oct 30 20:19:20.695876 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:20.695812    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podStartSLOduration=1.695794912 podStartE2EDuration="1.695794912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:20.695077982 +0000 UTC m=+893.598498040" watchObservedRunningTime="2026-10-30 20:19:20.695794912 +0000 UTC m=+893.599214939"
Oct 30 20:19:21.062815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:21.062741    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:23.464726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.464669    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"]
Oct 30 20:19:23.469188 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.469167    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.479707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.478245    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:23.479707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.478362    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:23.479707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.478425    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:23.479707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.478466    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:23.481197 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.481173    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"]
Oct 30 20:19:23.617590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.617557    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.617790 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.617768    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.617852 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.617816    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nc2\" (UniqueName: \"kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.719512 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.719432    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.719512 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.719473    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nc2\" (UniqueName: \"kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.719749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.719525    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.720176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.720150    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.721842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.721822    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.743563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.743540    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nc2\" (UniqueName: \"kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2\") pod \"maas-api-revoke-sub-10ae2c76-mw8tk\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:23.786192 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:23.786168    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:24.065219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.065190    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"]
Oct 30 20:19:24.066927 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:24.066900    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff95c6b9_d09b_45da_9f14_c00615e1ce40.slice/crio-ae519ef2319ca0b338b9c79b845a96e7f08d16ec7106a492b0d101576632702f WatchSource:0}: Error finding container ae519ef2319ca0b338b9c79b845a96e7f08d16ec7106a492b0d101576632702f: Status 404 returned error can't find the container with id ae519ef2319ca0b338b9c79b845a96e7f08d16ec7106a492b0d101576632702f
Oct 30 20:19:24.706387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.706297    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk" event={"ID":"ff95c6b9-d09b-45da-9f14-c00615e1ce40","Type":"ContainerStarted","Data":"26d55724660d1fb9c82d51e46cafdfab9e49abf0c3f3ec900d2ab5fbf8cd3ec5"}
Oct 30 20:19:24.706387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.706360    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk" event={"ID":"ff95c6b9-d09b-45da-9f14-c00615e1ce40","Type":"ContainerStarted","Data":"ae519ef2319ca0b338b9c79b845a96e7f08d16ec7106a492b0d101576632702f"}
Oct 30 20:19:24.727583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.727536    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk" podStartSLOduration=1.727523183 podStartE2EDuration="1.727523183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:24.725441139 +0000 UTC m=+897.628861166" watchObservedRunningTime="2026-10-30 20:19:24.727523183 +0000 UTC m=+897.630943208"
Oct 30 20:19:24.876684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.876642    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"]
Oct 30 20:19:24.881747 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.881724    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:24.894889 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.894856    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"]
Oct 30 20:19:24.996316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:24.996236    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"]
Oct 30 20:19:25.001197 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.001171    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.019095 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.019059    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"]
Oct 30 20:19:25.035949 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.035920    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.035955    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8j55\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036101    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036139    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036169    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036269 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036252    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036367    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036433 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036400    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.036621 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.036599    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.137924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.137893    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.137939    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138023    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138179 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138079    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbk4\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138179 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138169    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138191    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138211    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8j55\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138239    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138256    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138398    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138431    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138521 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138472    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138550    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138610    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138639    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138640    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138748    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138806    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.138828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138813    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138842    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.138944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138924    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.139070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.138944    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.139070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.139057    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.140556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.140527    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.140838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.140819    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.152800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.152772    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8j55\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.152944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.152924    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.200029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.199974    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.240903    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbk4\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.240985    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241076    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241151    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241212    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241255    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241285    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241389    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.241444    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.242223    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.242980    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.243297    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.244129    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.244453    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.250037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.245522    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.254702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.254663    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.257947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.257863    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.258436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.258173    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbk4\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4\") pod \"e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") " pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.323451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.323421    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:25.496200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.496172    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"]
Oct 30 20:19:25.497290 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:25.497264    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a608f1_c2a1_45a7_a933_e70386b5d7da.slice/crio-3915041624aaf9cd2370e8c153f98645502b6b5ca8a133337f98d0c9bbf26e2c WatchSource:0}: Error finding container 3915041624aaf9cd2370e8c153f98645502b6b5ca8a133337f98d0c9bbf26e2c: Status 404 returned error can't find the container with id 3915041624aaf9cd2370e8c153f98645502b6b5ca8a133337f98d0c9bbf26e2c
Oct 30 20:19:25.500368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.500300    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:25.500453 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.500399    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:25.500453 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.500438    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:25.620764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.620710    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"]
Oct 30 20:19:25.620895 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:25.620875    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c15f32_cdb3_4063_9335_ecf3e8818d64.slice/crio-2375f11f17b162deb4d2778a061cbd0fcf6cffd90acf0671c18e22586b042b05 WatchSource:0}: Error finding container 2375f11f17b162deb4d2778a061cbd0fcf6cffd90acf0671c18e22586b042b05: Status 404 returned error can't find the container with id 2375f11f17b162deb4d2778a061cbd0fcf6cffd90acf0671c18e22586b042b05
Oct 30 20:19:25.623366 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.623331    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:25.623475 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.623406    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:25.623475 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.623453    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:25.718709 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.718682    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" event={"ID":"37a608f1-c2a1-45a7-a933-e70386b5d7da","Type":"ContainerStarted","Data":"f1ca5781e504d243a217f96e5c4af2535d9c7141335650a656d7c928efa53b69"}
Oct 30 20:19:25.719142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.719103    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" event={"ID":"37a608f1-c2a1-45a7-a933-e70386b5d7da","Type":"ContainerStarted","Data":"3915041624aaf9cd2370e8c153f98645502b6b5ca8a133337f98d0c9bbf26e2c"}
Oct 30 20:19:25.721167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.721139    6058 generic.go:356] "Generic (PLEG): container finished" podID="ff95c6b9-d09b-45da-9f14-c00615e1ce40" containerID="26d55724660d1fb9c82d51e46cafdfab9e49abf0c3f3ec900d2ab5fbf8cd3ec5" exitCode=0
Oct 30 20:19:25.721265 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.721214    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk" event={"ID":"ff95c6b9-d09b-45da-9f14-c00615e1ce40","Type":"ContainerDied","Data":"26d55724660d1fb9c82d51e46cafdfab9e49abf0c3f3ec900d2ab5fbf8cd3ec5"}
Oct 30 20:19:25.723426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.723406    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" event={"ID":"98c15f32-cdb3-4063-9335-ecf3e8818d64","Type":"ContainerStarted","Data":"3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447"}
Oct 30 20:19:25.723561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.723543    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" event={"ID":"98c15f32-cdb3-4063-9335-ecf3e8818d64","Type":"ContainerStarted","Data":"2375f11f17b162deb4d2778a061cbd0fcf6cffd90acf0671c18e22586b042b05"}
Oct 30 20:19:25.775819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.772639    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podStartSLOduration=1.772621174 podStartE2EDuration="1.772621174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:25.746700277 +0000 UTC m=+898.650120302" watchObservedRunningTime="2026-10-30 20:19:25.772621174 +0000 UTC m=+898.676041200"
Oct 30 20:19:25.775819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:25.774573    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podStartSLOduration=1.774563981 podStartE2EDuration="1.774563981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:25.771038772 +0000 UTC m=+898.674458799" watchObservedRunningTime="2026-10-30 20:19:25.774563981 +0000 UTC m=+898.677984007"
Oct 30 20:19:26.200706 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:26.200674    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:26.201768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:26.201742    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:26.201912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:26.201786    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:26.325442 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:26.325409    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:26.325643 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:26.325499    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:26.325643 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:26.325578    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:27.233028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.218997    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:27.233028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.219105    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:27.418028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.399178    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:27.418028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.399256    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:27.626167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.626134    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:27.659392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.659334    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca\") pod \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") "
Oct 30 20:19:27.659559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.659474    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca\") pod \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") "
Oct 30 20:19:27.659681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.659662    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2\" (UniqueName: \"kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2\") pod \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\" (UID: \"ff95c6b9-d09b-45da-9f14-c00615e1ce40\") "
Oct 30 20:19:27.734850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.734803    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca" pod "ff95c6b9-d09b-45da-9f14-c00615e1ce40" (UID: "ff95c6b9-d09b-45da-9f14-c00615e1ce40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:27.756026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.736681    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca" pod "ff95c6b9-d09b-45da-9f14-c00615e1ce40" (UID: "ff95c6b9-d09b-45da-9f14-c00615e1ce40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:27.835285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.745260    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2" pod "ff95c6b9-d09b-45da-9f14-c00615e1ce40" (UID: "ff95c6b9-d09b-45da-9f14-c00615e1ce40"). InnerVolumeSpecName "kube-api-access-p2nc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:27.842185 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.789690    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:27.858955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.858923    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"
Oct 30 20:19:27.862894 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.859169    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk" event={"ID":"ff95c6b9-d09b-45da-9f14-c00615e1ce40","Type":"ContainerDied","Data":"ae519ef2319ca0b338b9c79b845a96e7f08d16ec7106a492b0d101576632702f"}
Oct 30 20:19:27.870164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.870141    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae519ef2319ca0b338b9c79b845a96e7f08d16ec7106a492b0d101576632702f"
Oct 30 20:19:27.945000 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.944914    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff95c6b9-d09b-45da-9f14-c00615e1ce40-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:27.968101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:27.968063    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2nc2\" (UniqueName: \"kubernetes.io/projected/ff95c6b9-d09b-45da-9f14-c00615e1ce40-kube-api-access-p2nc2\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:28.249549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.249459    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:28.250121 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.250085    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:28.281096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.281059    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"]
Oct 30 20:19:28.282414 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.282281    6058 cpu_manager.go:414] "containerMap: removing container" podUID="ff95c6b9-d09b-45da-9f14-c00615e1ce40" containerName="revoke-keys"
Oct 30 20:19:28.282595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.282575    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff95c6b9-d09b-45da-9f14-c00615e1ce40" containerName="revoke-keys"
Oct 30 20:19:28.282858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.282841    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff95c6b9-d09b-45da-9f14-c00615e1ce40" containerName="revoke-keys"
Oct 30 20:19:28.319040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.305254    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.338133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.332599    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:28.338133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.332660    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:28.339367 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.339330    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:28.340174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.340136    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:28.340284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.340251    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:28.340435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.340408    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:28.426058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.412670    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"]
Oct 30 20:19:28.547087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.538381    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.547087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.538514    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.547087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.538611    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwnb\" (UniqueName: \"kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.655028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.640519    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.655028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.640660    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.655028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.640765    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwnb\" (UniqueName: \"kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.655028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.644873    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.681029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.659272    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwnb\" (UniqueName: \"kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.685026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.682847    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cba0d32f-pptlw\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:28.731032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:28.730222    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:29.249178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.236418    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:29.249178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.248162    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:29.249683 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.249615    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:29.249683 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.249673    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:29.282028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.276099    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:29.282028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.277766    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:29.404080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.398800    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:29.439641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.438982    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:29.439641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.439077    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:29.792803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.791436    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:19:29.818121 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:29.818085    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"]
Oct 30 20:19:30.006586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.006539    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw" event={"ID":"630949d5-5f6e-4227-9aaa-5db2731dff90","Type":"ContainerStarted","Data":"9a44df6cb229075ebe8755b9bfa9f3a16930db249f2888c5bbd34c7876c42de1"}
Oct 30 20:19:30.065644 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.065612    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:30.100031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.086457    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:30.214347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.214313    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:30.214591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.214565    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:30.249164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.249114    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:30.250819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.250769    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:19:30.352087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.346132    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:30.352087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.346205    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:30.796031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.791688    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:30.796031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:30.792063    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:19:31.077089 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.073121    6058 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:31.101607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.092208    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:31.101607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.093079    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:19:31.141111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.124889    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:19:31.180195 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.146511    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw" event={"ID":"630949d5-5f6e-4227-9aaa-5db2731dff90","Type":"ContainerStarted","Data":"aaca24a1b2f4f59ad808f372ec11e7fd9ff0a86253430a9d9bc474dbcaa323d4"}
Oct 30 20:19:31.241146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.239804    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:31.241146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.239866    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:31.363070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.350291    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:31.363070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.350364    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:31.441352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.439831    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:19:31.441352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.440457    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:19:31.447905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:31.447853    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw" podStartSLOduration=4.447839161 podStartE2EDuration="4.447839161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:31.349905713 +0000 UTC m=+904.253325740" watchObservedRunningTime="2026-10-30 20:19:31.447839161 +0000 UTC m=+904.351259187"
Oct 30 20:19:31.451359 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:19:31.451320    6058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630949d5_5f6e_4227_9aaa_5db2731dff90.slice/crio-conmon-aaca24a1b2f4f59ad808f372ec11e7fd9ff0a86253430a9d9bc474dbcaa323d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:19:32.165691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:32.165651    6058 generic.go:356] "Generic (PLEG): container finished" podID="630949d5-5f6e-4227-9aaa-5db2731dff90" containerID="aaca24a1b2f4f59ad808f372ec11e7fd9ff0a86253430a9d9bc474dbcaa323d4" exitCode=0
Oct 30 20:19:32.166650 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:32.166617    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw" event={"ID":"630949d5-5f6e-4227-9aaa-5db2731dff90","Type":"ContainerDied","Data":"aaca24a1b2f4f59ad808f372ec11e7fd9ff0a86253430a9d9bc474dbcaa323d4"}
Oct 30 20:19:32.201342 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:32.201304    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:32.201671 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:32.201641    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:32.324381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:32.324342    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 20:19:32.324587 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:32.324413    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 20:19:33.201087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.201049    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:19:33.201546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.201117    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:19:33.441525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.441445    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:33.576016 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.575959    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb\" (UniqueName: \"kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb\") pod \"630949d5-5f6e-4227-9aaa-5db2731dff90\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") "
Oct 30 20:19:33.576187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.576084    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca\") pod \"630949d5-5f6e-4227-9aaa-5db2731dff90\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") "
Oct 30 20:19:33.576629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.576284    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca\") pod \"630949d5-5f6e-4227-9aaa-5db2731dff90\" (UID: \"630949d5-5f6e-4227-9aaa-5db2731dff90\") "
Oct 30 20:19:33.577136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.577109    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca" pod "630949d5-5f6e-4227-9aaa-5db2731dff90" (UID: "630949d5-5f6e-4227-9aaa-5db2731dff90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:33.579587 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.579548    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb" pod "630949d5-5f6e-4227-9aaa-5db2731dff90" (UID: "630949d5-5f6e-4227-9aaa-5db2731dff90"). InnerVolumeSpecName "kube-api-access-cgwnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:33.580326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.580291    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca" pod "630949d5-5f6e-4227-9aaa-5db2731dff90" (UID: "630949d5-5f6e-4227-9aaa-5db2731dff90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:33.682257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.682163    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgwnb\" (UniqueName: \"kubernetes.io/projected/630949d5-5f6e-4227-9aaa-5db2731dff90-kube-api-access-cgwnb\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:33.682257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.682198    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:33.682257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.682215    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630949d5-5f6e-4227-9aaa-5db2731dff90-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:33.834683 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.834583    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:33.835199 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.835155    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:33.848853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:33.846561    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:19:34.195502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:34.195464    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"
Oct 30 20:19:34.195755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:34.195728    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw" event={"ID":"630949d5-5f6e-4227-9aaa-5db2731dff90","Type":"ContainerDied","Data":"9a44df6cb229075ebe8755b9bfa9f3a16930db249f2888c5bbd34c7876c42de1"}
Oct 30 20:19:34.195895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:34.195878    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a44df6cb229075ebe8755b9bfa9f3a16930db249f2888c5bbd34c7876c42de1"
Oct 30 20:19:35.192024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:35.191980    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:35.192426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:35.192380    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:35.193352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:35.193334    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:19:40.076385 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:40.075595    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:40.093850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:40.093822    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:19:41.189156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.189123    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"]
Oct 30 20:19:41.189956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.189936    6058 cpu_manager.go:414] "containerMap: removing container" podUID="630949d5-5f6e-4227-9aaa-5db2731dff90" containerName="revoke-keys"
Oct 30 20:19:41.190029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.189959    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630949d5-5f6e-4227-9aaa-5db2731dff90" containerName="revoke-keys"
Oct 30 20:19:41.190158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.190146    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="630949d5-5f6e-4227-9aaa-5db2731dff90" containerName="revoke-keys"
Oct 30 20:19:41.194731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.194710    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.199803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.199760    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.200368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.200340    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.200482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.200390    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:41.200482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.200431    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:41.201676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.201653    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"]
Oct 30 20:19:41.289608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.289557    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrq2\" (UniqueName: \"kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.289831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.289744    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.289933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.289901    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.391283 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.391246    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrq2\" (UniqueName: \"kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.391443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.391313    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.391443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.391391    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.391958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.391935    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.393913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.393884    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.404536 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.404514    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrq2\" (UniqueName: \"kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2\") pod \"maas-api-revoke-sub-fe02ee94-plfhl\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.531751 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.531677    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:41.822375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:41.822350    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"]
Oct 30 20:19:41.823323 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:41.823293    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d62e78a_2a6b_4a62_a50c_d4bc4fced3f3.slice/crio-ed4ca3a7deeeb858e309faa00f107db944a5cccfb15dd67dad91eb5fff0f9896 WatchSource:0}: Error finding container ed4ca3a7deeeb858e309faa00f107db944a5cccfb15dd67dad91eb5fff0f9896: Status 404 returned error can't find the container with id ed4ca3a7deeeb858e309faa00f107db944a5cccfb15dd67dad91eb5fff0f9896
Oct 30 20:19:42.278055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.275562    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl" event={"ID":"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3","Type":"ContainerStarted","Data":"43c6a495190b8d8256a6588bc8a9f8e0ed3c8ddf0f9eec9fd52fdc1d96286ba7"}
Oct 30 20:19:42.278055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.275637    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl" event={"ID":"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3","Type":"ContainerStarted","Data":"ed4ca3a7deeeb858e309faa00f107db944a5cccfb15dd67dad91eb5fff0f9896"}
Oct 30 20:19:42.298505 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.298458    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl" podStartSLOduration=1.298444028 podStartE2EDuration="1.298444028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:42.294564012 +0000 UTC m=+915.197984038" watchObservedRunningTime="2026-10-30 20:19:42.298444028 +0000 UTC m=+915.201864054"
Oct 30 20:19:42.689411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.689381    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"]
Oct 30 20:19:42.693470 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.693450    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.704301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.704276    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:19:42.704398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.704298    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zcb7t\""
Oct 30 20:19:42.704398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.704276    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:19:42.704398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.704317    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:19:42.707959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.707939    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"]
Oct 30 20:19:42.714250 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714228    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.714355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714256    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.714355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714287    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.714506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714479    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rphx\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.714587 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714532    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.714651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714621    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.714708 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.714679    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815316    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rphx\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815360    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815402    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815427    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815467    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815614    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.815674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.815662    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.816141 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.816117    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.818054 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.817998    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.818136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.818106    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.818304 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.818274    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.819326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.819308    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.828186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.828163    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:42.828506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:42.828483    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rphx\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hm6jv\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:43.009657 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:43.009577    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:43.285719 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:43.285631    6058 generic.go:356] "Generic (PLEG): container finished" podID="0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" containerID="43c6a495190b8d8256a6588bc8a9f8e0ed3c8ddf0f9eec9fd52fdc1d96286ba7" exitCode=0
Oct 30 20:19:43.286129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:43.285717    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl" event={"ID":"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3","Type":"ContainerDied","Data":"43c6a495190b8d8256a6588bc8a9f8e0ed3c8ddf0f9eec9fd52fdc1d96286ba7"}
Oct 30 20:19:43.294497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:43.294471    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"]
Oct 30 20:19:43.296427 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:43.296388    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6955dd8c_7e85_4bbb_a1f5_2a59aeae8b5c.slice/crio-957e256d1efdf02e95ffeccfe8ef72bf97709aef2d7c679dc08406e5688d0186 WatchSource:0}: Error finding container 957e256d1efdf02e95ffeccfe8ef72bf97709aef2d7c679dc08406e5688d0186: Status 404 returned error can't find the container with id 957e256d1efdf02e95ffeccfe8ef72bf97709aef2d7c679dc08406e5688d0186
Oct 30 20:19:44.133367 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.133328    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:19:44.303900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.303862    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv" event={"ID":"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c","Type":"ContainerStarted","Data":"957e256d1efdf02e95ffeccfe8ef72bf97709aef2d7c679dc08406e5688d0186"}
Oct 30 20:19:44.456208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.456177    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:44.544753 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.544722    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca\") pod \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") "
Oct 30 20:19:44.544890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.544873    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca\") pod \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") "
Oct 30 20:19:44.544953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.544943    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2\" (UniqueName: \"kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2\") pod \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\" (UID: \"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3\") "
Oct 30 20:19:44.545260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.545234    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca" pod "0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" (UID: "0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:44.545482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.545462    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:44.546846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.546815    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca" pod "0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" (UID: "0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:44.546959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.546914    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2" pod "0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" (UID: "0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3"). InnerVolumeSpecName "kube-api-access-jbrq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:44.646987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.646951    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrq2\" (UniqueName: \"kubernetes.io/projected/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-kube-api-access-jbrq2\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:44.646987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:44.646979    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:45.317507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:45.317474    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"
Oct 30 20:19:45.317918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:45.317501    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl" event={"ID":"0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3","Type":"ContainerDied","Data":"ed4ca3a7deeeb858e309faa00f107db944a5cccfb15dd67dad91eb5fff0f9896"}
Oct 30 20:19:45.317918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:45.317564    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed4ca3a7deeeb858e309faa00f107db944a5cccfb15dd67dad91eb5fff0f9896"
Oct 30 20:19:47.162171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:47.162133    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:47.162462 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:47.162194    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:47.342040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:47.341971    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv" event={"ID":"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c","Type":"ContainerStarted","Data":"8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912"}
Oct 30 20:19:47.342229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:47.342104    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:47.372632 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:47.372577    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv" podStartSLOduration=1.510149541 podStartE2EDuration="5.372564766s" totalImagesPullingTime="3.862415225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:47.367508601 +0000 UTC m=+920.270928628" watchObservedRunningTime="2026-10-30 20:19:47.372564766 +0000 UTC m=+920.275984792"
Oct 30 20:19:48.350465 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:48.350433    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:19:49.658391 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:19:49.658348    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:19:54.089484 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.089447    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"]
Oct 30 20:19:54.090104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.090087    6058 cpu_manager.go:414] "containerMap: removing container" podUID="0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" containerName="revoke-keys"
Oct 30 20:19:54.090104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.090105    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" containerName="revoke-keys"
Oct 30 20:19:54.090235 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.090226    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" containerName="revoke-keys"
Oct 30 20:19:54.093744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.093725    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.100873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.100833    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:54.103321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.103193    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:54.103321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.103258    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:54.103321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.103280    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:54.103549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.103373    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"]
Oct 30 20:19:54.199743 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.199707    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s6d\" (UniqueName: \"kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.199903 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.199854    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.200037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.199993    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.301338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.301300    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.301530 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.301445    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.301530 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.301516    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s6d\" (UniqueName: \"kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.302146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.302117    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.303785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.303754    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.314409 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.314388    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s6d\" (UniqueName: \"kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d\") pod \"maas-api-revoke-sub-e76a8a92-m46bv\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.410162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.410124    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:54.712107 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:54.712077    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"]
Oct 30 20:19:54.712926 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:54.712893    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f166e04_09ad_4671_97b5_9c3ba29bb2d3.slice/crio-b9b090dcd6c2ea9da2daad4a6c299e8295055f7189bf5014da82b11306447d63 WatchSource:0}: Error finding container b9b090dcd6c2ea9da2daad4a6c299e8295055f7189bf5014da82b11306447d63: Status 404 returned error can't find the container with id b9b090dcd6c2ea9da2daad4a6c299e8295055f7189bf5014da82b11306447d63
Oct 30 20:19:55.177259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.177226    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"]
Oct 30 20:19:55.181588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.181563    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.225791    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.225887    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.225967    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.226111    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.226187    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.226773    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.226842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.226824    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxhl\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.227404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.227196    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.227404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.227267    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.228857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.228837    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"]
Oct 30 20:19:55.329054 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.328926    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329054 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329035    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329305 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329125    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329305 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329208    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329541    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329604    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329632    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329680    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxhl\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329761    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.329896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.329820    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.332376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.331269    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.332376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.332078    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.333344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.333302    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.333993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.333933    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.335041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.334988    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.340574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.340534    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.345206 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.345170    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxhl\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.347398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.347373    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token\") pod \"e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") " pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.405458 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.405421    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv" event={"ID":"7f166e04-09ad-4671-97b5-9c3ba29bb2d3","Type":"ContainerStarted","Data":"af1dd9b16900368ab87876316ab955e12f163ac15d567bacb8f8d9554b2790e5"}
Oct 30 20:19:55.405649 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.405481    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv" event={"ID":"7f166e04-09ad-4671-97b5-9c3ba29bb2d3","Type":"ContainerStarted","Data":"b9b090dcd6c2ea9da2daad4a6c299e8295055f7189bf5014da82b11306447d63"}
Oct 30 20:19:55.427601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.427552    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv" podStartSLOduration=1.427538212 podStartE2EDuration="1.427538212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:55.424419339 +0000 UTC m=+928.327839365" watchObservedRunningTime="2026-10-30 20:19:55.427538212 +0000 UTC m=+928.330958275"
Oct 30 20:19:55.496524 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.496494    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:55.788801 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.788769    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"]
Oct 30 20:19:55.789546 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:19:55.789521    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8034d7be_519d_40b7_b608_c452f72a6acf.slice/crio-8e47914c88df72d8704fd9861e7ad65bc806b6acae0de92b397c4ff271be5ea3 WatchSource:0}: Error finding container 8e47914c88df72d8704fd9861e7ad65bc806b6acae0de92b397c4ff271be5ea3: Status 404 returned error can't find the container with id 8e47914c88df72d8704fd9861e7ad65bc806b6acae0de92b397c4ff271be5ea3
Oct 30 20:19:55.792127 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.792094    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:55.792222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.792159    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:55.792222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:55.792189    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:19:56.415608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.415565    6058 generic.go:356] "Generic (PLEG): container finished" podID="7f166e04-09ad-4671-97b5-9c3ba29bb2d3" containerID="af1dd9b16900368ab87876316ab955e12f163ac15d567bacb8f8d9554b2790e5" exitCode=0
Oct 30 20:19:56.416036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.415647    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv" event={"ID":"7f166e04-09ad-4671-97b5-9c3ba29bb2d3","Type":"ContainerDied","Data":"af1dd9b16900368ab87876316ab955e12f163ac15d567bacb8f8d9554b2790e5"}
Oct 30 20:19:56.417415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.417392    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" event={"ID":"8034d7be-519d-40b7-b608-c452f72a6acf","Type":"ContainerStarted","Data":"a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12"}
Oct 30 20:19:56.417538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.417436    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" event={"ID":"8034d7be-519d-40b7-b608-c452f72a6acf","Type":"ContainerStarted","Data":"8e47914c88df72d8704fd9861e7ad65bc806b6acae0de92b397c4ff271be5ea3"}
Oct 30 20:19:56.483029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.474273    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podStartSLOduration=1.474253797 podStartE2EDuration="1.474253797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:56.47230701 +0000 UTC m=+929.375727053" watchObservedRunningTime="2026-10-30 20:19:56.474253797 +0000 UTC m=+929.377673824"
Oct 30 20:19:56.497363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.497333    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:19:56.498908 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.498884    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:19:56.499060 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:56.498935    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:19:57.500545 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.500399    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:19:57.500545 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.500466    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:19:57.706358 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.706288    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:57.797588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.797538    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca\") pod \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") "
Oct 30 20:19:57.797798 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.797701    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d\" (UniqueName: \"kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d\") pod \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") "
Oct 30 20:19:57.797882 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.797825    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca\") pod \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\" (UID: \"7f166e04-09ad-4671-97b5-9c3ba29bb2d3\") "
Oct 30 20:19:57.800424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.800390    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca" pod "7f166e04-09ad-4671-97b5-9c3ba29bb2d3" (UID: "7f166e04-09ad-4671-97b5-9c3ba29bb2d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:57.801895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.801860    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d" pod "7f166e04-09ad-4671-97b5-9c3ba29bb2d3" (UID: "7f166e04-09ad-4671-97b5-9c3ba29bb2d3"). InnerVolumeSpecName "kube-api-access-27s6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:57.802160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.802135    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca" pod "7f166e04-09ad-4671-97b5-9c3ba29bb2d3" (UID: "7f166e04-09ad-4671-97b5-9c3ba29bb2d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:57.899402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.899362    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:57.899402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.899395    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:57.899402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:57.899406    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27s6d\" (UniqueName: \"kubernetes.io/projected/7f166e04-09ad-4671-97b5-9c3ba29bb2d3-kube-api-access-27s6d\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:58.433036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:58.432990    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"
Oct 30 20:19:58.433299 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:58.433018    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv" event={"ID":"7f166e04-09ad-4671-97b5-9c3ba29bb2d3","Type":"ContainerDied","Data":"b9b090dcd6c2ea9da2daad4a6c299e8295055f7189bf5014da82b11306447d63"}
Oct 30 20:19:58.433372 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:58.433327    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b090dcd6c2ea9da2daad4a6c299e8295055f7189bf5014da82b11306447d63"
Oct 30 20:19:58.497499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:58.497464    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:19:58.497651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:58.497561    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:19:59.399721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.399683    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"]
Oct 30 20:19:59.400329 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.400310    6058 cpu_manager.go:414] "containerMap: removing container" podUID="7f166e04-09ad-4671-97b5-9c3ba29bb2d3" containerName="revoke-keys"
Oct 30 20:19:59.400401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.400329    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f166e04-09ad-4671-97b5-9c3ba29bb2d3" containerName="revoke-keys"
Oct 30 20:19:59.400462 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.400445    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f166e04-09ad-4671-97b5-9c3ba29bb2d3" containerName="revoke-keys"
Oct 30 20:19:59.403322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.403301    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.410126 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.410081    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:59.410376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.410356    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:59.410958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.410486    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:19:59.410958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.410572    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:59.413993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.413969    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"]
Oct 30 20:19:59.497212 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.497169    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:19:59.497400 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.497219    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:19:59.524832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.524803    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.524991 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.524843    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.525083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.525067    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrthj\" (UniqueName: \"kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.626413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.626376    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.626642 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.626622    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.626865 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.626853    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrthj\" (UniqueName: \"kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.627037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.626985    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.630273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.630252    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.640374 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.640355    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrthj\" (UniqueName: \"kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj\") pod \"maas-api-revoke-sub-7b086870-bb7sb\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:19:59.717709 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:19:59.717602    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:20:00.085776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:00.085130    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"]
Oct 30 20:20:00.447601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:00.447569    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb" event={"ID":"ed361e74-3609-4e97-b768-f7d4d83c396c","Type":"ContainerStarted","Data":"a000e882cf596a4f7342f1c3bb4861145f48cb6436a780d128f7f10f002cf1ee"}
Oct 30 20:20:00.497711 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:00.497678    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:20:00.497872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:00.497726    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:20:01.485509 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:01.485206    6058 generic.go:356] "Generic (PLEG): container finished" podID="ed361e74-3609-4e97-b768-f7d4d83c396c" containerID="721da78f755802c7d991d4e7538420cfba909f0f5650c4d073d40938572a5c6c" exitCode=0
Oct 30 20:20:01.486037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:01.485523    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb" event={"ID":"ed361e74-3609-4e97-b768-f7d4d83c396c","Type":"ContainerDied","Data":"721da78f755802c7d991d4e7538420cfba909f0f5650c4d073d40938572a5c6c"}
Oct 30 20:20:01.496872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:01.496839    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:20:01.496962 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:01.496901    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:20:02.497799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.497754    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:20:02.498264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.497826    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 20:20:02.669187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.669158    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:20:02.783265 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.783181    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj\" (UniqueName: \"kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj\") pod \"ed361e74-3609-4e97-b768-f7d4d83c396c\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") "
Oct 30 20:20:02.783421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.783387    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca\") pod \"ed361e74-3609-4e97-b768-f7d4d83c396c\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") "
Oct 30 20:20:02.783518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.783494    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca\") pod \"ed361e74-3609-4e97-b768-f7d4d83c396c\" (UID: \"ed361e74-3609-4e97-b768-f7d4d83c396c\") "
Oct 30 20:20:02.783805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.783776    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca" pod "ed361e74-3609-4e97-b768-f7d4d83c396c" (UID: "ed361e74-3609-4e97-b768-f7d4d83c396c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:02.784157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.784128    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:02.786157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.786125    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca" pod "ed361e74-3609-4e97-b768-f7d4d83c396c" (UID: "ed361e74-3609-4e97-b768-f7d4d83c396c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:02.786278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.786261    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj" pod "ed361e74-3609-4e97-b768-f7d4d83c396c" (UID: "ed361e74-3609-4e97-b768-f7d4d83c396c"). InnerVolumeSpecName "kube-api-access-qrthj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:02.888119 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.885630    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrthj\" (UniqueName: \"kubernetes.io/projected/ed361e74-3609-4e97-b768-f7d4d83c396c-kube-api-access-qrthj\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:02.888119 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:02.885671    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed361e74-3609-4e97-b768-f7d4d83c396c-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:03.520081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:03.518874    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb" event={"ID":"ed361e74-3609-4e97-b768-f7d4d83c396c","Type":"ContainerDied","Data":"a000e882cf596a4f7342f1c3bb4861145f48cb6436a780d128f7f10f002cf1ee"}
Oct 30 20:20:03.520081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:03.518940    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a000e882cf596a4f7342f1c3bb4861145f48cb6436a780d128f7f10f002cf1ee"
Oct 30 20:20:03.520081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:03.519032    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"
Oct 30 20:20:03.540026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:03.532155    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:20:03.540026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:03.533149    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:20:03.570046 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:03.566416    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:20:04.992031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:04.974687    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"]
Oct 30 20:20:05.017030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.006049    6058 cpu_manager.go:414] "containerMap: removing container" podUID="ed361e74-3609-4e97-b768-f7d4d83c396c" containerName="revoke-keys"
Oct 30 20:20:05.017030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.006083    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed361e74-3609-4e97-b768-f7d4d83c396c" containerName="revoke-keys"
Oct 30 20:20:05.017030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.006291    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed361e74-3609-4e97-b768-f7d4d83c396c" containerName="revoke-keys"
Oct 30 20:20:05.102029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.099345    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"]
Oct 30 20:20:05.102029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.099519    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.111032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.106264    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:20:05.191060 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.131106    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:05.191241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.131146    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:05.198033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.131174    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:20:05.208031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.140114    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:05.208031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.140331    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:05.208031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.140420    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:05.208031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.174329    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:05.224034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.223128    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:05.315043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.295662    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.315043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.295849    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crkk\" (UniqueName: \"kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.315043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.295897    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.333679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.333265    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:05.436655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.436596    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:20:05.748040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.743245    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crkk\" (UniqueName: \"kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.748040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.743339    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.748040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.743550    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:05.800392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.800349    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:05.822249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.818844    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:20:05.880651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.880611    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:06.005101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:05.995385    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crkk\" (UniqueName: \"kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:06.033169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.033115    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:06.219031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.209602    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:06.219031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.210396    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:20:06.219031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.212625    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:06.219031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.212628    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:20:06.219031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.213093    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:20:06.219031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.213148    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:20:06.240261 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.240133    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:20:06.244180 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.241974    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:20:06.280082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.248438    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c00fd4b3-55zpx\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:06.280082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.193217    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:06.280082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.269438    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:20:06.284152 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.284119    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:20:06.284398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.284370    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:06.345177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.345126    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:06.383188 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.383151    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:20:06.441210 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.422973    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:06.459257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.459211    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:20:06.459890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.459864    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:20:06.828034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:06.826527    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:20:07.188222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.185883    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:07.188222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.187235    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:20:07.218925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.218889    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:20:07.323506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.245786    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:20:07.323506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.263510    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:07.323506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.281867    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:07.334570 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.334546    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:20:07.334821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.334806    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:20:07.399431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.397535    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:20:07.399431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.397596    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:20:07.413036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.403938    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:20:07.605249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:07.602450    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"]
Oct 30 20:20:08.216192 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:08.216110    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx" event={"ID":"9c1115fe-1333-41d7-b0ed-4acfa8789e80","Type":"ContainerStarted","Data":"fd1631d011fcd1b11931f0b4bf6324955b69f920e4fadb97e139ad1f19a9dfc6"}
Oct 30 20:20:08.216192 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:08.216165    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx" event={"ID":"9c1115fe-1333-41d7-b0ed-4acfa8789e80","Type":"ContainerStarted","Data":"e055d2a7afa0776d548eb1278dfb40213b7b4bc49ce7cb8d660f9da091bef3f2"}
Oct 30 20:20:08.237388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:08.237344    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx" podStartSLOduration=4.23733221 podStartE2EDuration="4.23733221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:08.235674998 +0000 UTC m=+941.139095025" watchObservedRunningTime="2026-10-30 20:20:08.23733221 +0000 UTC m=+941.140752235"
Oct 30 20:20:09.223846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:09.223813    6058 generic.go:356] "Generic (PLEG): container finished" podID="9c1115fe-1333-41d7-b0ed-4acfa8789e80" containerID="fd1631d011fcd1b11931f0b4bf6324955b69f920e4fadb97e139ad1f19a9dfc6" exitCode=0
Oct 30 20:20:09.224321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:09.223897    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx" event={"ID":"9c1115fe-1333-41d7-b0ed-4acfa8789e80","Type":"ContainerDied","Data":"fd1631d011fcd1b11931f0b4bf6324955b69f920e4fadb97e139ad1f19a9dfc6"}
Oct 30 20:20:10.381645 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.381620    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:10.547224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.547137    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca\") pod \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") "
Oct 30 20:20:10.547413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.547224    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca\") pod \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") "
Oct 30 20:20:10.547413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.547249    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk\" (UniqueName: \"kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk\") pod \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\" (UID: \"9c1115fe-1333-41d7-b0ed-4acfa8789e80\") "
Oct 30 20:20:10.547693 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.547660    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca" pod "9c1115fe-1333-41d7-b0ed-4acfa8789e80" (UID: "9c1115fe-1333-41d7-b0ed-4acfa8789e80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:10.547955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.547929    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:10.549639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.549614    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk" pod "9c1115fe-1333-41d7-b0ed-4acfa8789e80" (UID: "9c1115fe-1333-41d7-b0ed-4acfa8789e80"). InnerVolumeSpecName "kube-api-access-8crkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:10.550059 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.549998    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca" pod "9c1115fe-1333-41d7-b0ed-4acfa8789e80" (UID: "9c1115fe-1333-41d7-b0ed-4acfa8789e80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:10.649538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.649498    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1115fe-1333-41d7-b0ed-4acfa8789e80-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:10.649538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:10.649526    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8crkk\" (UniqueName: \"kubernetes.io/projected/9c1115fe-1333-41d7-b0ed-4acfa8789e80-kube-api-access-8crkk\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:11.246730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:11.246701    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"
Oct 30 20:20:11.246885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:11.246726    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx" event={"ID":"9c1115fe-1333-41d7-b0ed-4acfa8789e80","Type":"ContainerDied","Data":"e055d2a7afa0776d548eb1278dfb40213b7b4bc49ce7cb8d660f9da091bef3f2"}
Oct 30 20:20:11.246885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:11.246812    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e055d2a7afa0776d548eb1278dfb40213b7b4bc49ce7cb8d660f9da091bef3f2"
Oct 30 20:20:18.174815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.174774    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"]
Oct 30 20:20:18.175496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.175475    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9c1115fe-1333-41d7-b0ed-4acfa8789e80" containerName="revoke-keys"
Oct 30 20:20:18.175496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.175495    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1115fe-1333-41d7-b0ed-4acfa8789e80" containerName="revoke-keys"
Oct 30 20:20:18.175647 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.175598    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1115fe-1333-41d7-b0ed-4acfa8789e80" containerName="revoke-keys"
Oct 30 20:20:18.182043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.182018    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.188340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.188318    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:18.188546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.188521    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:18.188676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.188661    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:20:18.188846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.188832    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:18.192821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.192716    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"]
Oct 30 20:20:18.249629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.249599    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbts\" (UniqueName: \"kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.249748 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.249722    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.249880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.249864    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.351056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.350999    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.352861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.352781    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.353297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.353273    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbts\" (UniqueName: \"kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.353602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.353583    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.367700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.367666    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.371195 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.371170    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbts\" (UniqueName: \"kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts\") pod \"maas-api-revoke-sub-9b85c442-rtzhj\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.500926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.500846    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:18.792716 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.792687    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"]
Oct 30 20:20:18.793704 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:20:18.793676    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea350170_1c31_4533_bc57_f3ba1a819c2b.slice/crio-2714701dcd7d77120a405efd0cf45a9cacbb5ef73418453478ccf05f6180ada2 WatchSource:0}: Error finding container 2714701dcd7d77120a405efd0cf45a9cacbb5ef73418453478ccf05f6180ada2: Status 404 returned error can't find the container with id 2714701dcd7d77120a405efd0cf45a9cacbb5ef73418453478ccf05f6180ada2
Oct 30 20:20:18.964989 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:18.964955    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:20:19.096041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.095946    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"]
Oct 30 20:20:19.102804 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.102773    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.110074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.109993    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"]
Oct 30 20:20:19.168614 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.168581    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blr7n\" (UniqueName: \"kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.168786 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.168764    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.168910 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.168886    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.270276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.270244    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blr7n\" (UniqueName: \"kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.270720 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.270368    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.270720 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.270475    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.271205 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.271180    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.272725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.272703    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.284251 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.284227    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blr7n\" (UniqueName: \"kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n\") pod \"maas-api-revoke-sub-3e14f5b7-nmwnf\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.313342 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.313313    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj" event={"ID":"ea350170-1c31-4533-bc57-f3ba1a819c2b","Type":"ContainerStarted","Data":"e8a44507f961164c433b7fced6a58dbe3731438cf33f72fba8deabc3817403e0"}
Oct 30 20:20:19.313466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.313365    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj" event={"ID":"ea350170-1c31-4533-bc57-f3ba1a819c2b","Type":"ContainerStarted","Data":"2714701dcd7d77120a405efd0cf45a9cacbb5ef73418453478ccf05f6180ada2"}
Oct 30 20:20:19.333775 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.333725    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj" podStartSLOduration=1.333712609 podStartE2EDuration="1.333712609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:19.332340098 +0000 UTC m=+952.235760127" watchObservedRunningTime="2026-10-30 20:20:19.333712609 +0000 UTC m=+952.237132640"
Oct 30 20:20:19.430828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.430797    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:19.661690 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:20:19.661652    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:20:19.720625 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:20:19.720584    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe7b56_1cd5_4a18_84b8_d0f37b0173d5.slice/crio-30cf73391f1fbd7c1d6914ddd363612db26d2a3018b76a12cd809c1af98bcf3e WatchSource:0}: Error finding container 30cf73391f1fbd7c1d6914ddd363612db26d2a3018b76a12cd809c1af98bcf3e: Status 404 returned error can't find the container with id 30cf73391f1fbd7c1d6914ddd363612db26d2a3018b76a12cd809c1af98bcf3e
Oct 30 20:20:19.721149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:19.721127    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"]
Oct 30 20:20:20.328381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:20.328275    6058 generic.go:356] "Generic (PLEG): container finished" podID="ea350170-1c31-4533-bc57-f3ba1a819c2b" containerID="e8a44507f961164c433b7fced6a58dbe3731438cf33f72fba8deabc3817403e0" exitCode=0
Oct 30 20:20:20.328381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:20.328364    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj" event={"ID":"ea350170-1c31-4533-bc57-f3ba1a819c2b","Type":"ContainerDied","Data":"e8a44507f961164c433b7fced6a58dbe3731438cf33f72fba8deabc3817403e0"}
Oct 30 20:20:20.331037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:20.330829    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf" event={"ID":"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5","Type":"ContainerStarted","Data":"8cbbeed94598de35f3fa7db071a62a074d8e4b99c0402ec165a1a9a5cc482f65"}
Oct 30 20:20:20.331037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:20.330882    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf" event={"ID":"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5","Type":"ContainerStarted","Data":"30cf73391f1fbd7c1d6914ddd363612db26d2a3018b76a12cd809c1af98bcf3e"}
Oct 30 20:20:20.397117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:20.397028    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf" podStartSLOduration=1.396989164 podStartE2EDuration="1.396989164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:20.366576727 +0000 UTC m=+953.269996757" watchObservedRunningTime="2026-10-30 20:20:20.396989164 +0000 UTC m=+953.300409195"
Oct 30 20:20:21.454029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:21.450267    6058 generic.go:356] "Generic (PLEG): container finished" podID="9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" containerID="8cbbeed94598de35f3fa7db071a62a074d8e4b99c0402ec165a1a9a5cc482f65" exitCode=0
Oct 30 20:20:21.454029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:21.450799    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf" event={"ID":"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5","Type":"ContainerDied","Data":"8cbbeed94598de35f3fa7db071a62a074d8e4b99c0402ec165a1a9a5cc482f65"}
Oct 30 20:20:22.168651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.168620    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:22.307421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.307334    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts\" (UniqueName: \"kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts\") pod \"ea350170-1c31-4533-bc57-f3ba1a819c2b\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") "
Oct 30 20:20:22.307595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.307575    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca\") pod \"ea350170-1c31-4533-bc57-f3ba1a819c2b\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") "
Oct 30 20:20:22.307712 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.307691    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca\") pod \"ea350170-1c31-4533-bc57-f3ba1a819c2b\" (UID: \"ea350170-1c31-4533-bc57-f3ba1a819c2b\") "
Oct 30 20:20:22.308058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.308030    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca" pod "ea350170-1c31-4533-bc57-f3ba1a819c2b" (UID: "ea350170-1c31-4533-bc57-f3ba1a819c2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:22.308247 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.308230    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:22.310106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.310079    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca" pod "ea350170-1c31-4533-bc57-f3ba1a819c2b" (UID: "ea350170-1c31-4533-bc57-f3ba1a819c2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:22.310190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.310143    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts" pod "ea350170-1c31-4533-bc57-f3ba1a819c2b" (UID: "ea350170-1c31-4533-bc57-f3ba1a819c2b"). InnerVolumeSpecName "kube-api-access-kxbts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:22.409405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.409377    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbts\" (UniqueName: \"kubernetes.io/projected/ea350170-1c31-4533-bc57-f3ba1a819c2b-kube-api-access-kxbts\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:22.409405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.409403    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea350170-1c31-4533-bc57-f3ba1a819c2b-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:22.459564 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.459535    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"
Oct 30 20:20:22.459978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.459568    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj" event={"ID":"ea350170-1c31-4533-bc57-f3ba1a819c2b","Type":"ContainerDied","Data":"2714701dcd7d77120a405efd0cf45a9cacbb5ef73418453478ccf05f6180ada2"}
Oct 30 20:20:22.459978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.459615    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2714701dcd7d77120a405efd0cf45a9cacbb5ef73418453478ccf05f6180ada2"
Oct 30 20:20:22.624677 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.624641    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:22.712782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.712752    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n\" (UniqueName: \"kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n\") pod \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") "
Oct 30 20:20:22.712960 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.712836    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca\") pod \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") "
Oct 30 20:20:22.712960 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.712885    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca\") pod \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\" (UID: \"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5\") "
Oct 30 20:20:22.713478 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.713253    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca" pod "9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" (UID: "9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:22.713478 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.713440    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:22.715342 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.715303    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n" pod "9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" (UID: "9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5"). InnerVolumeSpecName "kube-api-access-blr7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:22.715572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.715546    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca" pod "9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" (UID: "9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:22.814982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.814945    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blr7n\" (UniqueName: \"kubernetes.io/projected/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-kube-api-access-blr7n\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:22.814982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:22.814984    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:23.467881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:23.467785    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"
Oct 30 20:20:23.467881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:23.467797    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf" event={"ID":"9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5","Type":"ContainerDied","Data":"30cf73391f1fbd7c1d6914ddd363612db26d2a3018b76a12cd809c1af98bcf3e"}
Oct 30 20:20:23.467881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:23.467871    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cf73391f1fbd7c1d6914ddd363612db26d2a3018b76a12cd809c1af98bcf3e"
Oct 30 20:20:24.994719 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.994684    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"]
Oct 30 20:20:24.995569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.995546    6058 cpu_manager.go:414] "containerMap: removing container" podUID="ea350170-1c31-4533-bc57-f3ba1a819c2b" containerName="revoke-keys"
Oct 30 20:20:24.995670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.995570    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea350170-1c31-4533-bc57-f3ba1a819c2b" containerName="revoke-keys"
Oct 30 20:20:24.995670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.995590    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" containerName="revoke-keys"
Oct 30 20:20:24.995670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.995598    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" containerName="revoke-keys"
Oct 30 20:20:24.995822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.995725    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea350170-1c31-4533-bc57-f3ba1a819c2b" containerName="revoke-keys"
Oct 30 20:20:24.995822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:24.995737    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" containerName="revoke-keys"
Oct 30 20:20:25.000540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.000517    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.018780 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.018751    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"]
Oct 30 20:20:25.154464 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154425    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.154663 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154508    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.154663 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154603    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.154663 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154642    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.154848 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154718    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.154848 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154802    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.154939 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.154848    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.155032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.155015    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfc7k\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.155109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.155095    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256313 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256234    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256313 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256294    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256369    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256396    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256423    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256465    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256522    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256570    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256642    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256702    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfc7k\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.256977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.256905    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.257123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.257056    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.257123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.257059    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.257464 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.257432    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.258878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.258852    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.259507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.259484    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.269696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.269669    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.269836 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.269815    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfc7k\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k\") pod \"e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") " pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.334569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.334546    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:25.628485 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:20:25.628445    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84c7cda_57bf_438c_9329_a337f1bdcc96.slice/crio-5d5c8ee1a09d761ca8baa5ec810472b1984bb52514b80e23fc057a66c5feef40 WatchSource:0}: Error finding container 5d5c8ee1a09d761ca8baa5ec810472b1984bb52514b80e23fc057a66c5feef40: Status 404 returned error can't find the container with id 5d5c8ee1a09d761ca8baa5ec810472b1984bb52514b80e23fc057a66c5feef40
Oct 30 20:20:25.629075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.629043    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"]
Oct 30 20:20:25.630999 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.630967    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:20:25.631084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.631059    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:20:25.631133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:25.631101    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:20:26.506325 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:26.506285    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" event={"ID":"d84c7cda-57bf-438c-9329-a337f1bdcc96","Type":"ContainerStarted","Data":"5f5245684aaea5a6f5b48f2c7ad58df8ff1d9a637b01da6ef5f035e744d9e400"}
Oct 30 20:20:26.506707 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:26.506364    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" event={"ID":"d84c7cda-57bf-438c-9329-a337f1bdcc96","Type":"ContainerStarted","Data":"5d5c8ee1a09d761ca8baa5ec810472b1984bb52514b80e23fc057a66c5feef40"}
Oct 30 20:20:26.535515 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:26.535461    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podStartSLOduration=2.535447486 podStartE2EDuration="2.535447486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:26.53131045 +0000 UTC m=+959.434730491" watchObservedRunningTime="2026-10-30 20:20:26.535447486 +0000 UTC m=+959.438867589"
Oct 30 20:20:27.335302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:27.335271    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:27.336576 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:27.336552    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:27.336701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:27.336599    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:28.335738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:28.335702    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:28.336172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:28.335753    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:29.335019 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:29.334983    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:29.335201 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:29.335048    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:30.335173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:30.335134    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:30.335582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:30.335203    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:31.345029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:31.337348    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:31.345029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:31.337410    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:32.132028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.117378    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:32.189032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.188299    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:32.295030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.276737    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:32.340422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.340385    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:32.344112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.344074    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:32.413652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.411243    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:32.413652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.412319    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:32.413652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.412599    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:20:32.875191 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:32.875143    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:33.119189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.119129    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:33.120167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.120144    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:20:33.302215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.302109    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:33.346047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.346017    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:20:33.359162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.359127    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:33.359421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.359393    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:33.359912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.359886    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:33.382320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.382280    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:33.403197 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.403095    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 20:20:33.422035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.421607    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:33.422035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.421938    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:20:33.486030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.441628    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:20:33.546125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.488761    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 30 20:20:33.736095 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.641295    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:20:33.736491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.736456    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:20:33.757892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.749517    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:20:33.757892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.749578    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:20:33.866029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:33.831080    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Oct 30 20:20:34.106556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.106437    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:34.163905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.163866    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:20:34.434110 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.342341    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:20:34.434932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.434887    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:20:34.558032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.533662    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:20:34.635070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.609773    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:20:34.847088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.635385    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.636279    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.770596    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.641324    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.724852    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.658752    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5qft9" podUID="75e73891-2ed1-4066-83f7-4f9a9d384f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.637199    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:20:34.938031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:34.637422    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:20:35.183613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.183565    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:35.184310 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.184277    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 20:20:35.187083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.184544    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:35.278048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.187557    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:35.341193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.341158    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:20:35.341493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.341462    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:35.350276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.350244    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:35.450068 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.436127    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:35.476694 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.476663    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:20:35.558792 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.550362    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:20:35.568932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.568905    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:20:35.636078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.631470    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:20:35.670596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.670475    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:20:35.701069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.695601    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:20:35.756383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.756295    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:20:35.758951 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:35.758310    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:20:36.371237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:36.371196    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:20:36.371411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:36.371248    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:37.156030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.155984    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:37.156991 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.156959    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:37.158073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.158050    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:20:37.255083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.254989    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"]
Oct 30 20:20:37.261755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.261730    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.266988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.266960    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:20:37.267610 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.267296    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:37.267743 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.267360    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:37.267831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.267414    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:37.270489 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.270460    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"]
Oct 30 20:20:37.334791 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.334754    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklvw\" (UniqueName: \"kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.335104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.335072    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.335379 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.335346    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.436902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.436820    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.436902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.436883    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.437139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.437047    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklvw\" (UniqueName: \"kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.437705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.437672    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.439748 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.439717    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.456717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.456689    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklvw\" (UniqueName: \"kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw\") pod \"maas-api-revoke-sub-4b2d9e36-gm4rn\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.581545 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.581515    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:37.890857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:37.890821    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"]
Oct 30 20:20:37.891043 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:20:37.890889    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b00dab7_7025_4f89_9f4a_97c6a41a7bbb.slice/crio-52ae14a6bcf8c721631017161ce7d995959321895c252249399145dfd38eb287 WatchSource:0}: Error finding container 52ae14a6bcf8c721631017161ce7d995959321895c252249399145dfd38eb287: Status 404 returned error can't find the container with id 52ae14a6bcf8c721631017161ce7d995959321895c252249399145dfd38eb287
Oct 30 20:20:38.367398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:38.367360    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn" event={"ID":"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb","Type":"ContainerStarted","Data":"a94c3c083febd1a0cc50ecc3950fc89c504b84ffa848db9721ee0578c410edfc"}
Oct 30 20:20:38.367773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:38.367425    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn" event={"ID":"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb","Type":"ContainerStarted","Data":"52ae14a6bcf8c721631017161ce7d995959321895c252249399145dfd38eb287"}
Oct 30 20:20:38.387080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:38.387036    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn" podStartSLOduration=1.387022265 podStartE2EDuration="1.387022265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:38.385992691 +0000 UTC m=+971.289412717" watchObservedRunningTime="2026-10-30 20:20:38.387022265 +0000 UTC m=+971.290442285"
Oct 30 20:20:38.997626 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:38.997594    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"]
Oct 30 20:20:39.003062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.003039    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.011904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.011751    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"]
Oct 30 20:20:39.164487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.164450    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.164671 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.164642    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.164733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.164719    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kntw4\" (UniqueName: \"kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.265687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.265656    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.265857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.265769    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.265857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.265811    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kntw4\" (UniqueName: \"kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.266282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.266260    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.268128 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.268108    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.279044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.279020    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kntw4\" (UniqueName: \"kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4\") pod \"maas-api-revoke-sub-c0636f71-9tvl9\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.321219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.321190    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:39.392819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.392551    6058 generic.go:356] "Generic (PLEG): container finished" podID="1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" containerID="a94c3c083febd1a0cc50ecc3950fc89c504b84ffa848db9721ee0578c410edfc" exitCode=0
Oct 30 20:20:39.394799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.394327    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn" event={"ID":"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb","Type":"ContainerDied","Data":"a94c3c083febd1a0cc50ecc3950fc89c504b84ffa848db9721ee0578c410edfc"}
Oct 30 20:20:39.605538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:39.605506    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"]
Oct 30 20:20:39.605826 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:20:39.605788    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b52bde_0263_42b9_b4de_81ee45550d08.slice/crio-3fe28303046da22ff1f1364382ec0db12da84e29da78e89e08efb121aa8abf5e WatchSource:0}: Error finding container 3fe28303046da22ff1f1364382ec0db12da84e29da78e89e08efb121aa8abf5e: Status 404 returned error can't find the container with id 3fe28303046da22ff1f1364382ec0db12da84e29da78e89e08efb121aa8abf5e
Oct 30 20:20:40.405280 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.405241    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9" event={"ID":"79b52bde-0263-42b9-b4de-81ee45550d08","Type":"ContainerStarted","Data":"65bf1e498a1fbcbcfa7f456728fa73989b3d0291b442695b02108123752cee8a"}
Oct 30 20:20:40.405683 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.405307    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9" event={"ID":"79b52bde-0263-42b9-b4de-81ee45550d08","Type":"ContainerStarted","Data":"3fe28303046da22ff1f1364382ec0db12da84e29da78e89e08efb121aa8abf5e"}
Oct 30 20:20:40.428462 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.428410    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9" podStartSLOduration=2.428396781 podStartE2EDuration="2.428396781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:40.42428006 +0000 UTC m=+973.327700088" watchObservedRunningTime="2026-10-30 20:20:40.428396781 +0000 UTC m=+973.331816784"
Oct 30 20:20:40.677735 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.677673    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:40.786733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.786700    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca\") pod \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") "
Oct 30 20:20:40.786908 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.786822    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca\") pod \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") "
Oct 30 20:20:40.786989 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.786970    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw\" (UniqueName: \"kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw\") pod \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\" (UID: \"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb\") "
Oct 30 20:20:40.787221 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.787190    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca" pod "1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" (UID: "1b00dab7-7025-4f89-9f4a-97c6a41a7bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:40.787572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.787551    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.789468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.789433    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw" pod "1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" (UID: "1b00dab7-7025-4f89-9f4a-97c6a41a7bbb"). InnerVolumeSpecName "kube-api-access-hklvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:40.789571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.789510    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca" pod "1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" (UID: "1b00dab7-7025-4f89-9f4a-97c6a41a7bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:40.888912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.888886    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hklvw\" (UniqueName: \"kubernetes.io/projected/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-kube-api-access-hklvw\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.888912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:40.888911    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:41.417678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:41.417642    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"
Oct 30 20:20:41.418183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:41.417674    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn" event={"ID":"1b00dab7-7025-4f89-9f4a-97c6a41a7bbb","Type":"ContainerDied","Data":"52ae14a6bcf8c721631017161ce7d995959321895c252249399145dfd38eb287"}
Oct 30 20:20:41.418183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:41.417728    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ae14a6bcf8c721631017161ce7d995959321895c252249399145dfd38eb287"
Oct 30 20:20:41.420298 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:41.420271    6058 generic.go:356] "Generic (PLEG): container finished" podID="79b52bde-0263-42b9-b4de-81ee45550d08" containerID="65bf1e498a1fbcbcfa7f456728fa73989b3d0291b442695b02108123752cee8a" exitCode=0
Oct 30 20:20:41.420438 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:41.420335    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9" event={"ID":"79b52bde-0263-42b9-b4de-81ee45550d08","Type":"ContainerDied","Data":"65bf1e498a1fbcbcfa7f456728fa73989b3d0291b442695b02108123752cee8a"}
Oct 30 20:20:42.578508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.578476    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:42.714608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.714531    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4\" (UniqueName: \"kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4\") pod \"79b52bde-0263-42b9-b4de-81ee45550d08\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") "
Oct 30 20:20:42.714769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.714672    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca\") pod \"79b52bde-0263-42b9-b4de-81ee45550d08\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") "
Oct 30 20:20:42.714833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.714771    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca\") pod \"79b52bde-0263-42b9-b4de-81ee45550d08\" (UID: \"79b52bde-0263-42b9-b4de-81ee45550d08\") "
Oct 30 20:20:42.715053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.715025    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca" pod "79b52bde-0263-42b9-b4de-81ee45550d08" (UID: "79b52bde-0263-42b9-b4de-81ee45550d08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:42.715373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.715347    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:42.717158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.717127    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4" pod "79b52bde-0263-42b9-b4de-81ee45550d08" (UID: "79b52bde-0263-42b9-b4de-81ee45550d08"). InnerVolumeSpecName "kube-api-access-kntw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:42.717281 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.717264    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca" pod "79b52bde-0263-42b9-b4de-81ee45550d08" (UID: "79b52bde-0263-42b9-b4de-81ee45550d08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:42.816569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.816541    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kntw4\" (UniqueName: \"kubernetes.io/projected/79b52bde-0263-42b9-b4de-81ee45550d08-kube-api-access-kntw4\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:42.816700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:42.816576    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b52bde-0263-42b9-b4de-81ee45550d08-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:43.338656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:43.338619    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:20:43.435257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:43.435232    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"
Oct 30 20:20:43.435418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:43.435274    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9" event={"ID":"79b52bde-0263-42b9-b4de-81ee45550d08","Type":"ContainerDied","Data":"3fe28303046da22ff1f1364382ec0db12da84e29da78e89e08efb121aa8abf5e"}
Oct 30 20:20:43.435418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:43.435325    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe28303046da22ff1f1364382ec0db12da84e29da78e89e08efb121aa8abf5e"
Oct 30 20:20:44.177627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.177583    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"]
Oct 30 20:20:44.178598 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.178560    6058 cpu_manager.go:414] "containerMap: removing container" podUID="1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" containerName="revoke-keys"
Oct 30 20:20:44.178598 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.178588    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" containerName="revoke-keys"
Oct 30 20:20:44.178794 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.178605    6058 cpu_manager.go:414] "containerMap: removing container" podUID="79b52bde-0263-42b9-b4de-81ee45550d08" containerName="revoke-keys"
Oct 30 20:20:44.178794 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.178613    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b52bde-0263-42b9-b4de-81ee45550d08" containerName="revoke-keys"
Oct 30 20:20:44.178916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.178818    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" containerName="revoke-keys"
Oct 30 20:20:44.178916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.178832    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b52bde-0263-42b9-b4de-81ee45550d08" containerName="revoke-keys"
Oct 30 20:20:44.183346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.183323    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.188618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.188491    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:44.188618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.188590    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:20:44.189384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.189276    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:44.189384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.189355    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:44.224436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.224313    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"]
Oct 30 20:20:44.339181 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.339137    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.339355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.339266    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn92\" (UniqueName: \"kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.339355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.339345    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.442425 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.442324    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.442706 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.442689    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn92\" (UniqueName: \"kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.442861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.442839    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.445305 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.444239    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.446574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.446555    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.458209 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.458185    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn92\" (UniqueName: \"kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92\") pod \"maas-api-revoke-sub-9801bdc1-mrhql\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:44.565028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:44.560104    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:45.831633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:45.783178    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:45.831633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:45.784243    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:46.167032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.166951    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"]
Oct 30 20:20:46.204034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.194119    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:46.471418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.470199    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:46.558316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.557107    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:46.566177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.565494    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:46.587039 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.584805    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:20:46.723107 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.720253    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:46.860123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.806493    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:20:46.860123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.806894    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:20:46.860123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.807560    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:20:46.860123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:46.807622    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:20:47.136822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.136726    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql" event={"ID":"9013f712-b991-47b6-8100-615475806ee8","Type":"ContainerStarted","Data":"2fcb7ca65babadce5d79a157138506af16e2efd097f4868a3dc55050bab68edf"}
Oct 30 20:20:47.201939 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.201891    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:47.203669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.203646    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:20:47.228031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.223425    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Oct 30 20:20:47.478977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.473979    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:47.558262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.558220    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:47.614029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.595118    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:20:47.614029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.595209    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:20:47.640030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.635086    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:20:47.640030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.636262    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:20:47.829129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.802228    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:47.829129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.802637    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:20:47.966345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.966290    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:20:47.979027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:47.975130    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:48.026681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.007731    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:20:48.125532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.125491    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:20:48.219371 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.219337    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql" event={"ID":"9013f712-b991-47b6-8100-615475806ee8","Type":"ContainerStarted","Data":"3362363d0717ffa337ed5bbff4c5d8e9b112aec51125c9b48cbb00d8307fbad9"}
Oct 30 20:20:48.376549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.376439    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:20:48.380101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.380081    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:20:48.382681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.382664    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:20:48.384360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.384342    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:20:48.394067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.388749    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:20:48.743623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:48.743561    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql" podStartSLOduration=4.743541082 podStartE2EDuration="4.743541082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:48.415315844 +0000 UTC m=+981.318735868" watchObservedRunningTime="2026-10-30 20:20:48.743541082 +0000 UTC m=+981.646961109"
Oct 30 20:20:48.848031 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:20:48.843944    6058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9013f712_b991_47b6_8100_615475806ee8.slice/crio-conmon-3362363d0717ffa337ed5bbff4c5d8e9b112aec51125c9b48cbb00d8307fbad9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9013f712_b991_47b6_8100_615475806ee8.slice/crio-3362363d0717ffa337ed5bbff4c5d8e9b112aec51125c9b48cbb00d8307fbad9.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:20:49.226393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:49.226352    6058 generic.go:356] "Generic (PLEG): container finished" podID="9013f712-b991-47b6-8100-615475806ee8" containerID="3362363d0717ffa337ed5bbff4c5d8e9b112aec51125c9b48cbb00d8307fbad9" exitCode=0
Oct 30 20:20:49.226749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:49.226397    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql" event={"ID":"9013f712-b991-47b6-8100-615475806ee8","Type":"ContainerDied","Data":"3362363d0717ffa337ed5bbff4c5d8e9b112aec51125c9b48cbb00d8307fbad9"}
Oct 30 20:20:49.659303 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:20:49.659263    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:20:50.423938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.423910    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:50.503988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.503917    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca\") pod \"9013f712-b991-47b6-8100-615475806ee8\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") "
Oct 30 20:20:50.504149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.504022    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca\") pod \"9013f712-b991-47b6-8100-615475806ee8\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") "
Oct 30 20:20:50.504207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.504168    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92\" (UniqueName: \"kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92\") pod \"9013f712-b991-47b6-8100-615475806ee8\" (UID: \"9013f712-b991-47b6-8100-615475806ee8\") "
Oct 30 20:20:50.504334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.504309    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca" pod "9013f712-b991-47b6-8100-615475806ee8" (UID: "9013f712-b991-47b6-8100-615475806ee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:50.504819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.504802    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9013f712-b991-47b6-8100-615475806ee8-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.505992 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.505971    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca" pod "9013f712-b991-47b6-8100-615475806ee8" (UID: "9013f712-b991-47b6-8100-615475806ee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:50.506139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.506122    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92" pod "9013f712-b991-47b6-8100-615475806ee8" (UID: "9013f712-b991-47b6-8100-615475806ee8"). InnerVolumeSpecName "kube-api-access-9cn92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:50.606723 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.606693    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cn92\" (UniqueName: \"kubernetes.io/projected/9013f712-b991-47b6-8100-615475806ee8-kube-api-access-9cn92\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.606723 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:50.606723    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9013f712-b991-47b6-8100-615475806ee8-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:51.270578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.270550    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"
Oct 30 20:20:51.270760 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.270572    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql" event={"ID":"9013f712-b991-47b6-8100-615475806ee8","Type":"ContainerDied","Data":"2fcb7ca65babadce5d79a157138506af16e2efd097f4868a3dc55050bab68edf"}
Oct 30 20:20:51.270760 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.270661    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fcb7ca65babadce5d79a157138506af16e2efd097f4868a3dc55050bab68edf"
Oct 30 20:20:51.478125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.478088    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"]
Oct 30 20:20:51.479043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.479021    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9013f712-b991-47b6-8100-615475806ee8" containerName="revoke-keys"
Oct 30 20:20:51.479088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.479046    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9013f712-b991-47b6-8100-615475806ee8" containerName="revoke-keys"
Oct 30 20:20:51.479227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.479213    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9013f712-b991-47b6-8100-615475806ee8" containerName="revoke-keys"
Oct 30 20:20:51.483774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.483751    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.491521 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.491487    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:51.492833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.492433    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"]
Oct 30 20:20:51.492961 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.492505    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:51.492961 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.492605    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:20:51.492961 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.492660    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:51.624245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.624215    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.624398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.624251    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mztmm\" (UniqueName: \"kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.624398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.624374    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.725451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.725425    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.725451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.725457    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mztmm\" (UniqueName: \"kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.725706 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.725520    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.726267 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.726241    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.727924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.727896    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.739220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.739197    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mztmm\" (UniqueName: \"kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm\") pod \"maas-api-revoke-sub-769d6ada-ltvrq\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:51.804508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:51.804470    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:52.088464 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:52.088434    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"]
Oct 30 20:20:52.088600 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:20:52.088430    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179272ea_cfa3_474e_bf19_b34cef4a9bc1.slice/crio-3883fb125dec6f11dde602f3c5725e2f7083126b297fa655563fd8d3c08fb6b5 WatchSource:0}: Error finding container 3883fb125dec6f11dde602f3c5725e2f7083126b297fa655563fd8d3c08fb6b5: Status 404 returned error can't find the container with id 3883fb125dec6f11dde602f3c5725e2f7083126b297fa655563fd8d3c08fb6b5
Oct 30 20:20:52.280544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:52.280511    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq" event={"ID":"179272ea-cfa3-474e-bf19-b34cef4a9bc1","Type":"ContainerStarted","Data":"3883fb125dec6f11dde602f3c5725e2f7083126b297fa655563fd8d3c08fb6b5"}
Oct 30 20:20:53.289959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:53.289923    6058 generic.go:356] "Generic (PLEG): container finished" podID="179272ea-cfa3-474e-bf19-b34cef4a9bc1" containerID="36331a4c857fe962f3f518d9f0c4140aa0d4584a4f2f028521a7ac023bc90064" exitCode=0
Oct 30 20:20:53.290449 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:53.290034    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq" event={"ID":"179272ea-cfa3-474e-bf19-b34cef4a9bc1","Type":"ContainerDied","Data":"36331a4c857fe962f3f518d9f0c4140aa0d4584a4f2f028521a7ac023bc90064"}
Oct 30 20:20:54.466403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.466376    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:54.574622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.574543    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca\") pod \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") "
Oct 30 20:20:54.574776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.574705    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca\") pod \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") "
Oct 30 20:20:54.574847 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.574832    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm\" (UniqueName: \"kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm\") pod \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\" (UID: \"179272ea-cfa3-474e-bf19-b34cef4a9bc1\") "
Oct 30 20:20:54.575105 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.575079    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca" pod "179272ea-cfa3-474e-bf19-b34cef4a9bc1" (UID: "179272ea-cfa3-474e-bf19-b34cef4a9bc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:54.575276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.575262    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:54.576656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.576632    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca" pod "179272ea-cfa3-474e-bf19-b34cef4a9bc1" (UID: "179272ea-cfa3-474e-bf19-b34cef4a9bc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:54.576803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.576786    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm" pod "179272ea-cfa3-474e-bf19-b34cef4a9bc1" (UID: "179272ea-cfa3-474e-bf19-b34cef4a9bc1"). InnerVolumeSpecName "kube-api-access-mztmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:54.680056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.677700    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mztmm\" (UniqueName: \"kubernetes.io/projected/179272ea-cfa3-474e-bf19-b34cef4a9bc1-kube-api-access-mztmm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:54.680056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:54.677737    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179272ea-cfa3-474e-bf19-b34cef4a9bc1-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:55.307494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:55.307454    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"
Oct 30 20:20:55.307494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:55.307477    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq" event={"ID":"179272ea-cfa3-474e-bf19-b34cef4a9bc1","Type":"ContainerDied","Data":"3883fb125dec6f11dde602f3c5725e2f7083126b297fa655563fd8d3c08fb6b5"}
Oct 30 20:20:55.307692 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:20:55.307536    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3883fb125dec6f11dde602f3c5725e2f7083126b297fa655563fd8d3c08fb6b5"
Oct 30 20:21:02.980071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:02.974070    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:21:03.037050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:03.036467    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:03.617637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:03.617597    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:21:03.617935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:03.617901    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:03.812121 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:03.811784    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:07.607702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.607624    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"]
Oct 30 20:21:07.608512 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.608488    6058 cpu_manager.go:414] "containerMap: removing container" podUID="179272ea-cfa3-474e-bf19-b34cef4a9bc1" containerName="revoke-keys"
Oct 30 20:21:07.608583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.608515    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179272ea-cfa3-474e-bf19-b34cef4a9bc1" containerName="revoke-keys"
Oct 30 20:21:07.608713 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.608699    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="179272ea-cfa3-474e-bf19-b34cef4a9bc1" containerName="revoke-keys"
Oct 30 20:21:07.614104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.614080    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.619430 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.619405    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:07.619554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.619512    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:07.619963 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.619945    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:07.620147 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.619971    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"]
Oct 30 20:21:07.620487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.620434    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:07.634670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.634643    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.634791 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.634682    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkxh\" (UniqueName: \"kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.634874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.634846    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.736256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.736216    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.736540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.736515    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.736658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.736568    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkxh\" (UniqueName: \"kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.737510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.737483    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.739700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.739669    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.751626 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.751605    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkxh\" (UniqueName: \"kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh\") pod \"maas-api-revoke-sub-409e39e9-4kcqf\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:07.932555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:07.932476    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:08.264818 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:08.264779    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04c7537_7eea_41b6_965e_efeaa585c8aa.slice/crio-e606bdd4709d58e02d04b5d80e7b6f641dcde7252097fa62a9a36b56a1b9ce6e WatchSource:0}: Error finding container e606bdd4709d58e02d04b5d80e7b6f641dcde7252097fa62a9a36b56a1b9ce6e: Status 404 returned error can't find the container with id e606bdd4709d58e02d04b5d80e7b6f641dcde7252097fa62a9a36b56a1b9ce6e
Oct 30 20:21:08.265078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:08.265050    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"]
Oct 30 20:21:08.450296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:08.450261    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf" event={"ID":"a04c7537-7eea-41b6-965e-efeaa585c8aa","Type":"ContainerStarted","Data":"e606bdd4709d58e02d04b5d80e7b6f641dcde7252097fa62a9a36b56a1b9ce6e"}
Oct 30 20:21:09.462151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:09.462064    6058 generic.go:356] "Generic (PLEG): container finished" podID="a04c7537-7eea-41b6-965e-efeaa585c8aa" containerID="4b168548d8178cac89c57eb229cc025612e7cd6b50ca7f1e7b1ce7c6d620d76c" exitCode=0
Oct 30 20:21:09.462582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:09.462150    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf" event={"ID":"a04c7537-7eea-41b6-965e-efeaa585c8aa","Type":"ContainerDied","Data":"4b168548d8178cac89c57eb229cc025612e7cd6b50ca7f1e7b1ce7c6d620d76c"}
Oct 30 20:21:10.622083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.622060    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:10.683762 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.683733    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca\") pod \"a04c7537-7eea-41b6-965e-efeaa585c8aa\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") "
Oct 30 20:21:10.683944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.683838    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh\" (UniqueName: \"kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh\") pod \"a04c7537-7eea-41b6-965e-efeaa585c8aa\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") "
Oct 30 20:21:10.684231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.684205    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca" pod "a04c7537-7eea-41b6-965e-efeaa585c8aa" (UID: "a04c7537-7eea-41b6-965e-efeaa585c8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:10.684421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.684399    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca\") pod \"a04c7537-7eea-41b6-965e-efeaa585c8aa\" (UID: \"a04c7537-7eea-41b6-965e-efeaa585c8aa\") "
Oct 30 20:21:10.685055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.685031    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:10.686547 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.686518    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh" pod "a04c7537-7eea-41b6-965e-efeaa585c8aa" (UID: "a04c7537-7eea-41b6-965e-efeaa585c8aa"). InnerVolumeSpecName "kube-api-access-lmkxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:10.687065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.687037    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca" pod "a04c7537-7eea-41b6-965e-efeaa585c8aa" (UID: "a04c7537-7eea-41b6-965e-efeaa585c8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:10.787138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.787071    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkxh\" (UniqueName: \"kubernetes.io/projected/a04c7537-7eea-41b6-965e-efeaa585c8aa-kube-api-access-lmkxh\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:10.787138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:10.787107    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04c7537-7eea-41b6-965e-efeaa585c8aa-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:11.480020 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:11.479973    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"
Oct 30 20:21:11.480211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:11.479981    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf" event={"ID":"a04c7537-7eea-41b6-965e-efeaa585c8aa","Type":"ContainerDied","Data":"e606bdd4709d58e02d04b5d80e7b6f641dcde7252097fa62a9a36b56a1b9ce6e"}
Oct 30 20:21:11.480211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:11.480075    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e606bdd4709d58e02d04b5d80e7b6f641dcde7252097fa62a9a36b56a1b9ce6e"
Oct 30 20:21:16.310061 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:16.310025    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:21:18.286029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.269263    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"]
Oct 30 20:21:18.286705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.286364    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a04c7537-7eea-41b6-965e-efeaa585c8aa" containerName="revoke-keys"
Oct 30 20:21:18.286705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.286390    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04c7537-7eea-41b6-965e-efeaa585c8aa" containerName="revoke-keys"
Oct 30 20:21:18.286705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.286622    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04c7537-7eea-41b6-965e-efeaa585c8aa" containerName="revoke-keys"
Oct 30 20:21:18.296028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.294279    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.306030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.301102    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"]
Oct 30 20:21:18.321027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.315767    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:18.326026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.323224    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:18.326026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.323307    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:18.326026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.323348    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:18.468053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.432964    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlw4\" (UniqueName: \"kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.468053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.433157    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.468053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.433273    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.548257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.548162    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.672031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.641386    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.675080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.675043    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlw4\" (UniqueName: \"kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.675862 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.565258    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.694705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.694674    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.739729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.739692    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlw4\" (UniqueName: \"kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4\") pod \"maas-api-revoke-sub-5365edb9-ck2gg\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:18.994980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:18.994946    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:19.148029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.127163    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:19.359863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.359819    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 20:21:19.381603 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.381557    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:21:19.431041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.427445    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:19.612930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.593200    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:21:19.612930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.593270    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:19.688085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.675378    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:21:19.843680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.787515    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:19.933095 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.909155    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:19.996916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.951974    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:21:19.996916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:19.952060    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:20.085145 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:20.085106    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:20.199428 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:21:20.199346    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:21:20.240074 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:20.217449    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b53f23_768a_464c_8486_a2e38b90629d.slice/crio-3f3ebb183be98db6d1874da3d7612546a378f502207fb7a8ab6992b59c545809 WatchSource:0}: Error finding container 3f3ebb183be98db6d1874da3d7612546a378f502207fb7a8ab6992b59c545809: Status 404 returned error can't find the container with id 3f3ebb183be98db6d1874da3d7612546a378f502207fb7a8ab6992b59c545809
Oct 30 20:21:20.287288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:20.284312    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"]
Oct 30 20:21:20.918510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:20.918469    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg" event={"ID":"c5b53f23-768a-464c-8486-a2e38b90629d","Type":"ContainerStarted","Data":"d5bdac4892fd54fdcc90e352c0626d64537f0c21699a0449b796dfac612a9883"}
Oct 30 20:21:20.918884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:20.918558    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg" event={"ID":"c5b53f23-768a-464c-8486-a2e38b90629d","Type":"ContainerStarted","Data":"3f3ebb183be98db6d1874da3d7612546a378f502207fb7a8ab6992b59c545809"}
Oct 30 20:21:20.940226 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:20.940175    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg" podStartSLOduration=2.940162042 podStartE2EDuration="2.940162042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:20.9375479 +0000 UTC m=+1013.840967926" watchObservedRunningTime="2026-10-30 20:21:20.940162042 +0000 UTC m=+1013.843582068"
Oct 30 20:21:21.684919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.684886    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"]
Oct 30 20:21:21.689487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.689462    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.701348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.701315    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"]
Oct 30 20:21:21.736052 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.735995    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.736259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.736056    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtskg\" (UniqueName: \"kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.736491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.736466    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.837901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.837857    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.838740 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.838710    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.839174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.839141    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.839296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.839201    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtskg\" (UniqueName: \"kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.842266 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.842241    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.853043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.853000    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtskg\" (UniqueName: \"kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg\") pod \"maas-api-revoke-sub-6fc1b223-thcrg\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:21.927692 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.927657    6058 generic.go:356] "Generic (PLEG): container finished" podID="c5b53f23-768a-464c-8486-a2e38b90629d" containerID="d5bdac4892fd54fdcc90e352c0626d64537f0c21699a0449b796dfac612a9883" exitCode=0
Oct 30 20:21:21.928090 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:21.927757    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg" event={"ID":"c5b53f23-768a-464c-8486-a2e38b90629d","Type":"ContainerDied","Data":"d5bdac4892fd54fdcc90e352c0626d64537f0c21699a0449b796dfac612a9883"}
Oct 30 20:21:22.007226 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.007154    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:22.295964 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.295932    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"]
Oct 30 20:21:22.296224 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:22.296180    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf05cc8_c8e7_4a08_b7f1_c1ef5983f07c.slice/crio-254985120b9175f0c4c3dd25b73a76183499d7919c2785864fc7914e1359922b WatchSource:0}: Error finding container 254985120b9175f0c4c3dd25b73a76183499d7919c2785864fc7914e1359922b: Status 404 returned error can't find the container with id 254985120b9175f0c4c3dd25b73a76183499d7919c2785864fc7914e1359922b
Oct 30 20:21:22.676296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.676269    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"]
Oct 30 20:21:22.680530 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.680510    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.694627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.694603    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"]
Oct 30 20:21:22.752563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.752522    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.752871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.752833    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsssh\" (UniqueName: \"kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.753057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.752980    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.855376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.855340    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsssh\" (UniqueName: \"kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.855555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.855409    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.855555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.855551    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.856193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.856165    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.857758 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.857736    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.868367 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.868330    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsssh\" (UniqueName: \"kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh\") pod \"maas-api-revoke-sub-1582d838-vfn24\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:22.939140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.939048    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg" event={"ID":"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c","Type":"ContainerStarted","Data":"2be5ade4ccf13b13b804e2c2397201a872b3b000fe73c70fb95a6c046ea330b6"}
Oct 30 20:21:22.939140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.939127    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg" event={"ID":"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c","Type":"ContainerStarted","Data":"254985120b9175f0c4c3dd25b73a76183499d7919c2785864fc7914e1359922b"}
Oct 30 20:21:22.961078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.960988    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg" podStartSLOduration=1.9609703490000001 podStartE2EDuration="1.960970349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:22.959100146 +0000 UTC m=+1015.862520174" watchObservedRunningTime="2026-10-30 20:21:22.960970349 +0000 UTC m=+1015.864390375"
Oct 30 20:21:22.995136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:22.995104    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:23.126974 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.126936    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:23.163028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.160560    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4\" (UniqueName: \"kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4\") pod \"c5b53f23-768a-464c-8486-a2e38b90629d\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") "
Oct 30 20:21:23.163028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.160677    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca\") pod \"c5b53f23-768a-464c-8486-a2e38b90629d\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") "
Oct 30 20:21:23.163028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.160797    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca\") pod \"c5b53f23-768a-464c-8486-a2e38b90629d\" (UID: \"c5b53f23-768a-464c-8486-a2e38b90629d\") "
Oct 30 20:21:23.163028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.162150    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca" pod "c5b53f23-768a-464c-8486-a2e38b90629d" (UID: "c5b53f23-768a-464c-8486-a2e38b90629d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:23.164537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.163534    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca" pod "c5b53f23-768a-464c-8486-a2e38b90629d" (UID: "c5b53f23-768a-464c-8486-a2e38b90629d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:23.164537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.163948    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4" pod "c5b53f23-768a-464c-8486-a2e38b90629d" (UID: "c5b53f23-768a-464c-8486-a2e38b90629d"). InnerVolumeSpecName "kube-api-access-szlw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:23.263218 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.263190    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:23.263312 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.263221    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szlw4\" (UniqueName: \"kubernetes.io/projected/c5b53f23-768a-464c-8486-a2e38b90629d-kube-api-access-szlw4\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:23.263312 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.263237    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b53f23-768a-464c-8486-a2e38b90629d-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:23.320240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.320205    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"]
Oct 30 20:21:23.949920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.949855    6058 generic.go:356] "Generic (PLEG): container finished" podID="adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" containerID="2be5ade4ccf13b13b804e2c2397201a872b3b000fe73c70fb95a6c046ea330b6" exitCode=0
Oct 30 20:21:23.950371 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.949934    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg" event={"ID":"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c","Type":"ContainerDied","Data":"2be5ade4ccf13b13b804e2c2397201a872b3b000fe73c70fb95a6c046ea330b6"}
Oct 30 20:21:23.952069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.952039    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24" event={"ID":"9857d9c1-7d1d-45e1-8340-076f34f189c3","Type":"ContainerStarted","Data":"596b9223f19566ed59f13612713e9a12e75dd6fdcee9de89393de7daac49e371"}
Oct 30 20:21:23.952227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.952091    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24" event={"ID":"9857d9c1-7d1d-45e1-8340-076f34f189c3","Type":"ContainerStarted","Data":"2233e015729693e5c761d62b34892a8d8699c8fccaf84db28cd7ecd657cf03f7"}
Oct 30 20:21:23.953550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.953525    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg" event={"ID":"c5b53f23-768a-464c-8486-a2e38b90629d","Type":"ContainerDied","Data":"3f3ebb183be98db6d1874da3d7612546a378f502207fb7a8ab6992b59c545809"}
Oct 30 20:21:23.953696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.953558    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"
Oct 30 20:21:23.953696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:23.953568    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3ebb183be98db6d1874da3d7612546a378f502207fb7a8ab6992b59c545809"
Oct 30 20:21:24.038758 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:24.038711    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24" podStartSLOduration=2.03869915 podStartE2EDuration="2.03869915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:24.035650962 +0000 UTC m=+1016.939070990" watchObservedRunningTime="2026-10-30 20:21:24.03869915 +0000 UTC m=+1016.942119207"
Oct 30 20:21:24.963527 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:24.963489    6058 generic.go:356] "Generic (PLEG): container finished" podID="9857d9c1-7d1d-45e1-8340-076f34f189c3" containerID="596b9223f19566ed59f13612713e9a12e75dd6fdcee9de89393de7daac49e371" exitCode=0
Oct 30 20:21:24.963993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:24.963586    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24" event={"ID":"9857d9c1-7d1d-45e1-8340-076f34f189c3","Type":"ContainerDied","Data":"596b9223f19566ed59f13612713e9a12e75dd6fdcee9de89393de7daac49e371"}
Oct 30 20:21:25.132776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.132747    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:25.187464 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.187434    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg\" (UniqueName: \"kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg\") pod \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") "
Oct 30 20:21:25.187619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.187484    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca\") pod \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") "
Oct 30 20:21:25.187828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.187802    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca\") pod \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\" (UID: \"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c\") "
Oct 30 20:21:25.188267 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.188241    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca" pod "adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" (UID: "adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:25.188523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.188501    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:25.190404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.190374    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg" pod "adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" (UID: "adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c"). InnerVolumeSpecName "kube-api-access-qtskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:25.190689 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.190654    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca" pod "adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" (UID: "adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:25.289625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.289590    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtskg\" (UniqueName: \"kubernetes.io/projected/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-kube-api-access-qtskg\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:25.289625 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.289621    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:25.974987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.974957    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"
Oct 30 20:21:25.975472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.974985    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg" event={"ID":"adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c","Type":"ContainerDied","Data":"254985120b9175f0c4c3dd25b73a76183499d7919c2785864fc7914e1359922b"}
Oct 30 20:21:25.975472 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:25.975071    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254985120b9175f0c4c3dd25b73a76183499d7919c2785864fc7914e1359922b"
Oct 30 20:21:26.178053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.174360    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:26.304098 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.303983    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca\") pod \"9857d9c1-7d1d-45e1-8340-076f34f189c3\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") "
Oct 30 20:21:26.304252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.304234    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca\") pod \"9857d9c1-7d1d-45e1-8340-076f34f189c3\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") "
Oct 30 20:21:26.304315 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.304304    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh\" (UniqueName: \"kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh\") pod \"9857d9c1-7d1d-45e1-8340-076f34f189c3\" (UID: \"9857d9c1-7d1d-45e1-8340-076f34f189c3\") "
Oct 30 20:21:26.304801 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.304570    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca" pod "9857d9c1-7d1d-45e1-8340-076f34f189c3" (UID: "9857d9c1-7d1d-45e1-8340-076f34f189c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:26.304962 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.304938    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:26.306239 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.306214    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca" pod "9857d9c1-7d1d-45e1-8340-076f34f189c3" (UID: "9857d9c1-7d1d-45e1-8340-076f34f189c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:26.306435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.306411    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh" pod "9857d9c1-7d1d-45e1-8340-076f34f189c3" (UID: "9857d9c1-7d1d-45e1-8340-076f34f189c3"). InnerVolumeSpecName "kube-api-access-bsssh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:26.406034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.405994    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9857d9c1-7d1d-45e1-8340-076f34f189c3-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:26.406034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.406035    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsssh\" (UniqueName: \"kubernetes.io/projected/9857d9c1-7d1d-45e1-8340-076f34f189c3-kube-api-access-bsssh\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:26.987326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.987290    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"
Oct 30 20:21:26.987808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.987331    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24" event={"ID":"9857d9c1-7d1d-45e1-8340-076f34f189c3","Type":"ContainerDied","Data":"2233e015729693e5c761d62b34892a8d8699c8fccaf84db28cd7ecd657cf03f7"}
Oct 30 20:21:26.987808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:26.987432    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2233e015729693e5c761d62b34892a8d8699c8fccaf84db28cd7ecd657cf03f7"
Oct 30 20:21:27.130246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.130207    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"]
Oct 30 20:21:27.131531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131502    6058 cpu_manager.go:414] "containerMap: removing container" podUID="c5b53f23-768a-464c-8486-a2e38b90629d" containerName="revoke-keys"
Oct 30 20:21:27.131531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131530    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b53f23-768a-464c-8486-a2e38b90629d" containerName="revoke-keys"
Oct 30 20:21:27.131724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131551    6058 cpu_manager.go:414] "containerMap: removing container" podUID="adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" containerName="revoke-keys"
Oct 30 20:21:27.131724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131558    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" containerName="revoke-keys"
Oct 30 20:21:27.131724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131623    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9857d9c1-7d1d-45e1-8340-076f34f189c3" containerName="revoke-keys"
Oct 30 20:21:27.131724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131631    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9857d9c1-7d1d-45e1-8340-076f34f189c3" containerName="revoke-keys"
Oct 30 20:21:27.131938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131901    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9857d9c1-7d1d-45e1-8340-076f34f189c3" containerName="revoke-keys"
Oct 30 20:21:27.131938 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131934    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b53f23-768a-464c-8486-a2e38b90629d" containerName="revoke-keys"
Oct 30 20:21:27.132035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.131943    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" containerName="revoke-keys"
Oct 30 20:21:27.137187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.137156    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.143085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.142544    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:27.143085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.142616    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:27.143259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.143173    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:27.143322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.143269    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:27.144855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.144833    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"]
Oct 30 20:21:27.216142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.216111    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dg4z\" (UniqueName: \"kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.216298 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.216182    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.216381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.216301    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.317256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.317180    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.317403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.317382    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dg4z\" (UniqueName: \"kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.317514 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.317497    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.320157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.320137    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:27.320157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.320143    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:27.328586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.328565    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.330198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.330180    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.330751 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.330736    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:27.341253 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.341234    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dg4z\" (UniqueName: \"kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z\") pod \"maas-api-revoke-sub-a9ac179e-dfw6d\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.458628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.458602    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:27.466158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.466127    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:27.765500 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:27.755141    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8f7190_e22e_4e8d_b1c5_13b1b6e87762.slice/crio-c4fef11a54b07b3518bcecbc9cc332b099531b016f0e548208626242b31bf234 WatchSource:0}: Error finding container c4fef11a54b07b3518bcecbc9cc332b099531b016f0e548208626242b31bf234: Status 404 returned error can't find the container with id c4fef11a54b07b3518bcecbc9cc332b099531b016f0e548208626242b31bf234
Oct 30 20:21:27.765500 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.755320    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"]
Oct 30 20:21:27.997360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:27.997320    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d" event={"ID":"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762","Type":"ContainerStarted","Data":"c4fef11a54b07b3518bcecbc9cc332b099531b016f0e548208626242b31bf234"}
Oct 30 20:21:29.006182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:29.006143    6058 generic.go:356] "Generic (PLEG): container finished" podID="9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" containerID="1de8f8561bd2a6b66e0f87c7091385972bdb454414979b6f777eec0bdc52414a" exitCode=0
Oct 30 20:21:29.006561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:29.006220    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d" event={"ID":"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762","Type":"ContainerDied","Data":"1de8f8561bd2a6b66e0f87c7091385972bdb454414979b6f777eec0bdc52414a"}
Oct 30 20:21:30.680324 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.680290    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:30.696060 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.696030    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca\") pod \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") "
Oct 30 20:21:30.791365 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.791328    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca" pod "9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" (UID: "9d8f7190-e22e-4e8d-b1c5-13b1b6e87762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:30.805484 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.805450    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca\") pod \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") "
Oct 30 20:21:30.805931 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.805903    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z\" (UniqueName: \"kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z\") pod \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\" (UID: \"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762\") "
Oct 30 20:21:30.806874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.806837    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.816387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.816352    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca" pod "9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" (UID: "9d8f7190-e22e-4e8d-b1c5-13b1b6e87762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:30.844103 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.835550    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z" pod "9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" (UID: "9d8f7190-e22e-4e8d-b1c5-13b1b6e87762"). InnerVolumeSpecName "kube-api-access-4dg4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:30.935067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.934964    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dg4z\" (UniqueName: \"kubernetes.io/projected/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-kube-api-access-4dg4z\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:30.935280 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:30.935260    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:31.093029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.090362    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d" event={"ID":"9d8f7190-e22e-4e8d-b1c5-13b1b6e87762","Type":"ContainerDied","Data":"c4fef11a54b07b3518bcecbc9cc332b099531b016f0e548208626242b31bf234"}
Oct 30 20:21:31.093029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.090461    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4fef11a54b07b3518bcecbc9cc332b099531b016f0e548208626242b31bf234"
Oct 30 20:21:31.093029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.090535    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"
Oct 30 20:21:31.386026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.377281    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:21:31.544079 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.538077    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"]
Oct 30 20:21:31.544079 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.538988    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" containerName="revoke-keys"
Oct 30 20:21:31.613099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.613056    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" containerName="revoke-keys"
Oct 30 20:21:31.613606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.613579    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" containerName="revoke-keys"
Oct 30 20:21:31.671615 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.671583    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"]
Oct 30 20:21:31.677627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.672212    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.738519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.694408    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:31.738519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.694527    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:31.738519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.694596    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:31.738519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.697157    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:31.824089 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.795805    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:31.828871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.828833    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.861033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.859991    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.861033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.860141    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5mz\" (UniqueName: \"kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.962912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.962867    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.963240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.963214    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5mz\" (UniqueName: \"kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.974034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.963468    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.975640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.975611    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:31.989227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:31.989180    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:32.015369 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.013091    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:32.022241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.019400    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:32.051036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.049168    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5mz\" (UniqueName: \"kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz\") pod \"maas-api-revoke-sub-0dc831e2-stpld\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:32.182494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.182460    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:32.183978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.183942    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 20:21:32.380145 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.380099    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:32.381163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.381137    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:21:32.583088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.555092    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:21:32.647641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.647593    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:32.673035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.672191    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:21:32.862975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.832928    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:32.862975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:32.834125    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:21:33.113087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.086935    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:21:33.113087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.088314    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:21:33.236029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.139594    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:21:33.236029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.182157    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:21:33.236029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.192078    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 20:21:33.236029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.192436    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:21:33.451146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.381085    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.452799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.452758    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.564030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.563108    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.564030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.563627    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:21:33.682589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.652105    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:21:33.682589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.652175    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.682589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.652781    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.826146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.817104    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:21:33.826146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.817177    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.932042 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.921160    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:33.932759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:33.932677    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.014034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.013973    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.121062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.090108    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.170122 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.148528    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.198523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.198099    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.334028 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:21:34.289215    6058 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.015s"
Oct 30 20:21:34.334357 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.334330    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:21:34.353676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.353648    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:21:34.379028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.353954    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:21:34.379306 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.379284    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:21:34.400041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.390452    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.736141 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.714697    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:21:34.915024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.910787    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:21:34.915024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.910895    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:34.915024 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.913837    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:21:34.915358 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.915336    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:21:34.915583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.915567    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:21:34.915748 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.915735    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:21:34.927300 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.927278    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:21:34.943175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.943153    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:21:34.946419 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.946388    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:21:34.978952 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:34.978895    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"]
Oct 30 20:21:35.278270 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:35.278225    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld" event={"ID":"a9f9abdb-d115-4fb5-aa0c-bdeac194a503","Type":"ContainerStarted","Data":"0a32b8e074416a0cd08a0714144c5ff23ddae6441ea5afc5303548720073a625"}
Oct 30 20:21:36.286828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:36.286791    6058 generic.go:356] "Generic (PLEG): container finished" podID="a9f9abdb-d115-4fb5-aa0c-bdeac194a503" containerID="ef6268aa97cfe0a7465a0d66c2a2e19a1cca36d136a52bd70db02315fd14cb8b" exitCode=0
Oct 30 20:21:36.287336 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:36.286846    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld" event={"ID":"a9f9abdb-d115-4fb5-aa0c-bdeac194a503","Type":"ContainerDied","Data":"ef6268aa97cfe0a7465a0d66c2a2e19a1cca36d136a52bd70db02315fd14cb8b"}
Oct 30 20:21:37.377328 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.377295    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"]
Oct 30 20:21:37.394558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.394533    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.397777 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.397752    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"]
Oct 30 20:21:37.485751 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.485722    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:37.557582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.557498    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.557582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.557551    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.557857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.557702    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpxb\" (UniqueName: \"kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.658288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.658252    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz\" (UniqueName: \"kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz\") pod \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") "
Oct 30 20:21:37.658483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.658401    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca\") pod \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") "
Oct 30 20:21:37.658904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.658546    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca\") pod \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\" (UID: \"a9f9abdb-d115-4fb5-aa0c-bdeac194a503\") "
Oct 30 20:21:37.658904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.658860    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.659133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.658925    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.659133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.658983    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpxb\" (UniqueName: \"kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.660231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.660180    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca" pod "a9f9abdb-d115-4fb5-aa0c-bdeac194a503" (UID: "a9f9abdb-d115-4fb5-aa0c-bdeac194a503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:37.660674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.660629    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.681307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.679152    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz" pod "a9f9abdb-d115-4fb5-aa0c-bdeac194a503" (UID: "a9f9abdb-d115-4fb5-aa0c-bdeac194a503"). InnerVolumeSpecName "kube-api-access-kn5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:37.681307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.680570    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpxb\" (UniqueName: \"kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.683582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.683556    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca" pod "a9f9abdb-d115-4fb5-aa0c-bdeac194a503" (UID: "a9f9abdb-d115-4fb5-aa0c-bdeac194a503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:37.684500 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.684455    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4929e599-67wch\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.718496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.718462    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:37.760668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.760632    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:37.760806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.760675    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:37.760806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:37.760694    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn5mz\" (UniqueName: \"kubernetes.io/projected/a9f9abdb-d115-4fb5-aa0c-bdeac194a503-kube-api-access-kn5mz\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:38.002928 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:38.002887    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289ded7b_1109_4a03_8b4a_703ad84ce11c.slice/crio-b98c96bc90ad1f0811b4a23c0c666f9c516fedb7856efb4880a6a78464673dc5 WatchSource:0}: Error finding container b98c96bc90ad1f0811b4a23c0c666f9c516fedb7856efb4880a6a78464673dc5: Status 404 returned error can't find the container with id b98c96bc90ad1f0811b4a23c0c666f9c516fedb7856efb4880a6a78464673dc5
Oct 30 20:21:38.003132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.003020    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"]
Oct 30 20:21:38.310443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.310412    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch" event={"ID":"289ded7b-1109-4a03-8b4a-703ad84ce11c","Type":"ContainerStarted","Data":"b184411df43e9fa11c3829198a4fc71dbe2bd13219050b57982f93e1485c583f"}
Oct 30 20:21:38.310621 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.310474    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch" event={"ID":"289ded7b-1109-4a03-8b4a-703ad84ce11c","Type":"ContainerStarted","Data":"b98c96bc90ad1f0811b4a23c0c666f9c516fedb7856efb4880a6a78464673dc5"}
Oct 30 20:21:38.311915 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.311891    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"
Oct 30 20:21:38.312036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.311932    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld" event={"ID":"a9f9abdb-d115-4fb5-aa0c-bdeac194a503","Type":"ContainerDied","Data":"0a32b8e074416a0cd08a0714144c5ff23ddae6441ea5afc5303548720073a625"}
Oct 30 20:21:38.312036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.311988    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a32b8e074416a0cd08a0714144c5ff23ddae6441ea5afc5303548720073a625"
Oct 30 20:21:38.334412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.334360    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch" podStartSLOduration=1.334346386 podStartE2EDuration="1.334346386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:38.328937761 +0000 UTC m=+1031.232357786" watchObservedRunningTime="2026-10-30 20:21:38.334346386 +0000 UTC m=+1031.237766412"
Oct 30 20:21:38.960967 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.960937    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"]
Oct 30 20:21:38.961540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.961526    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a9f9abdb-d115-4fb5-aa0c-bdeac194a503" containerName="revoke-keys"
Oct 30 20:21:38.961591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.961541    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f9abdb-d115-4fb5-aa0c-bdeac194a503" containerName="revoke-keys"
Oct 30 20:21:38.961643 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.961634    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f9abdb-d115-4fb5-aa0c-bdeac194a503" containerName="revoke-keys"
Oct 30 20:21:38.965174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.965157    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:38.975897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:38.975871    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"]
Oct 30 20:21:39.075780 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.075698    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.075922 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.075883    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.075970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.075924    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtjp\" (UniqueName: \"kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.177186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.177156    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtjp\" (UniqueName: \"kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.177376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.177273    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.177473 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.177453    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.178064 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.178031    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.179902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.179878    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.191225 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.191203    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtjp\" (UniqueName: \"kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp\") pod \"maas-api-revoke-sub-3e7af2aa-pnmxv\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.281588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.281558    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:39.346574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.346188    6058 generic.go:356] "Generic (PLEG): container finished" podID="289ded7b-1109-4a03-8b4a-703ad84ce11c" containerID="b184411df43e9fa11c3829198a4fc71dbe2bd13219050b57982f93e1485c583f" exitCode=0
Oct 30 20:21:39.346574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.346393    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch" event={"ID":"289ded7b-1109-4a03-8b4a-703ad84ce11c","Type":"ContainerDied","Data":"b184411df43e9fa11c3829198a4fc71dbe2bd13219050b57982f93e1485c583f"}
Oct 30 20:21:39.605794 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:39.605749    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode971d5d5_5122_42d1_bf0b_62eea12b4196.slice/crio-bcfca3584ec35b5eefcd92b74cb7d5b3b9600b605ea13229f34e18646873d183 WatchSource:0}: Error finding container bcfca3584ec35b5eefcd92b74cb7d5b3b9600b605ea13229f34e18646873d183: Status 404 returned error can't find the container with id bcfca3584ec35b5eefcd92b74cb7d5b3b9600b605ea13229f34e18646873d183
Oct 30 20:21:39.606038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:39.605982    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"]
Oct 30 20:21:40.356731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.356638    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv" event={"ID":"e971d5d5-5122-42d1-bf0b-62eea12b4196","Type":"ContainerStarted","Data":"cd00e8c757bc4e1727be7ce3412e9ae2b971170352bff6ad12213e741d19cad8"}
Oct 30 20:21:40.356731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.356698    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv" event={"ID":"e971d5d5-5122-42d1-bf0b-62eea12b4196","Type":"ContainerStarted","Data":"bcfca3584ec35b5eefcd92b74cb7d5b3b9600b605ea13229f34e18646873d183"}
Oct 30 20:21:40.379656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.379580    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv" podStartSLOduration=2.379561474 podStartE2EDuration="2.379561474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:40.376136636 +0000 UTC m=+1033.279556673" watchObservedRunningTime="2026-10-30 20:21:40.379561474 +0000 UTC m=+1033.282981502"
Oct 30 20:21:40.520848 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.520819    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:40.707382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.707354    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca\") pod \"289ded7b-1109-4a03-8b4a-703ad84ce11c\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") "
Oct 30 20:21:40.707382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.707388    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca\") pod \"289ded7b-1109-4a03-8b4a-703ad84ce11c\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") "
Oct 30 20:21:40.707577 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.707487    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb\" (UniqueName: \"kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb\") pod \"289ded7b-1109-4a03-8b4a-703ad84ce11c\" (UID: \"289ded7b-1109-4a03-8b4a-703ad84ce11c\") "
Oct 30 20:21:40.707746 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.707718    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca" pod "289ded7b-1109-4a03-8b4a-703ad84ce11c" (UID: "289ded7b-1109-4a03-8b4a-703ad84ce11c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:40.708058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.708035    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:40.709496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.709467    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb" pod "289ded7b-1109-4a03-8b4a-703ad84ce11c" (UID: "289ded7b-1109-4a03-8b4a-703ad84ce11c"). InnerVolumeSpecName "kube-api-access-zgpxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:40.709595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.709529    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca" pod "289ded7b-1109-4a03-8b4a-703ad84ce11c" (UID: "289ded7b-1109-4a03-8b4a-703ad84ce11c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:40.809177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.809151    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpxb\" (UniqueName: \"kubernetes.io/projected/289ded7b-1109-4a03-8b4a-703ad84ce11c-kube-api-access-zgpxb\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:40.809177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:40.809176    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289ded7b-1109-4a03-8b4a-703ad84ce11c-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:41.368871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:41.368834    6058 generic.go:356] "Generic (PLEG): container finished" podID="e971d5d5-5122-42d1-bf0b-62eea12b4196" containerID="cd00e8c757bc4e1727be7ce3412e9ae2b971170352bff6ad12213e741d19cad8" exitCode=0
Oct 30 20:21:41.369496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:41.368938    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv" event={"ID":"e971d5d5-5122-42d1-bf0b-62eea12b4196","Type":"ContainerDied","Data":"cd00e8c757bc4e1727be7ce3412e9ae2b971170352bff6ad12213e741d19cad8"}
Oct 30 20:21:41.372210 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:41.372180    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch" event={"ID":"289ded7b-1109-4a03-8b4a-703ad84ce11c","Type":"ContainerDied","Data":"b98c96bc90ad1f0811b4a23c0c666f9c516fedb7856efb4880a6a78464673dc5"}
Oct 30 20:21:41.372360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:41.372213    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"
Oct 30 20:21:41.372360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:41.372232    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98c96bc90ad1f0811b4a23c0c666f9c516fedb7856efb4880a6a78464673dc5"
Oct 30 20:21:42.547702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.547666    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:42.633429 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.633297    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca\") pod \"e971d5d5-5122-42d1-bf0b-62eea12b4196\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") "
Oct 30 20:21:42.633589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.633538    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca\") pod \"e971d5d5-5122-42d1-bf0b-62eea12b4196\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") "
Oct 30 20:21:42.633640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.633600    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp\" (UniqueName: \"kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp\") pod \"e971d5d5-5122-42d1-bf0b-62eea12b4196\" (UID: \"e971d5d5-5122-42d1-bf0b-62eea12b4196\") "
Oct 30 20:21:42.635129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.634741    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca" pod "e971d5d5-5122-42d1-bf0b-62eea12b4196" (UID: "e971d5d5-5122-42d1-bf0b-62eea12b4196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:42.636980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.636952    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca" pod "e971d5d5-5122-42d1-bf0b-62eea12b4196" (UID: "e971d5d5-5122-42d1-bf0b-62eea12b4196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:42.638715 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.638681    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp" pod "e971d5d5-5122-42d1-bf0b-62eea12b4196" (UID: "e971d5d5-5122-42d1-bf0b-62eea12b4196"). InnerVolumeSpecName "kube-api-access-rwtjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:42.735116 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.735087    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:42.735116 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.735114    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e971d5d5-5122-42d1-bf0b-62eea12b4196-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:42.735301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:42.735124    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtjp\" (UniqueName: \"kubernetes.io/projected/e971d5d5-5122-42d1-bf0b-62eea12b4196-kube-api-access-rwtjp\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:43.392256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.392217    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv" event={"ID":"e971d5d5-5122-42d1-bf0b-62eea12b4196","Type":"ContainerDied","Data":"bcfca3584ec35b5eefcd92b74cb7d5b3b9600b605ea13229f34e18646873d183"}
Oct 30 20:21:43.392256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.392251    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"
Oct 30 20:21:43.392499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.392312    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcfca3584ec35b5eefcd92b74cb7d5b3b9600b605ea13229f34e18646873d183"
Oct 30 20:21:43.414042 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414016    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"]
Oct 30 20:21:43.414592 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414578    6058 cpu_manager.go:414] "containerMap: removing container" podUID="e971d5d5-5122-42d1-bf0b-62eea12b4196" containerName="revoke-keys"
Oct 30 20:21:43.414638 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414593    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e971d5d5-5122-42d1-bf0b-62eea12b4196" containerName="revoke-keys"
Oct 30 20:21:43.414638 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414623    6058 cpu_manager.go:414] "containerMap: removing container" podUID="289ded7b-1109-4a03-8b4a-703ad84ce11c" containerName="revoke-keys"
Oct 30 20:21:43.414638 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414628    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289ded7b-1109-4a03-8b4a-703ad84ce11c" containerName="revoke-keys"
Oct 30 20:21:43.414740 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414730    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="289ded7b-1109-4a03-8b4a-703ad84ce11c" containerName="revoke-keys"
Oct 30 20:21:43.414784 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.414747    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e971d5d5-5122-42d1-bf0b-62eea12b4196" containerName="revoke-keys"
Oct 30 20:21:43.418599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.418581    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.426346 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.426324    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:43.426490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.426436    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:43.426930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.426587    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:43.426930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.426724    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:43.428177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.428155    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"]
Oct 30 20:21:43.447075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.447046    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.447491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.447355    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.447491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.447414    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.549058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.548968    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.549388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.549161    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.549388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.549190    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.549773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.549740    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.551435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.551412    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.562513 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.562489    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8\") pod \"maas-api-revoke-sub-04e0db5c-7jsdj\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:43.736469 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:43.736436    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:44.035435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:44.035407    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"]
Oct 30 20:21:44.402466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:44.402431    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj" event={"ID":"165323da-d482-48d7-8616-0f20cef73c1f","Type":"ContainerStarted","Data":"19e0784c7584e6bc704b42bc75d1feb524fac7c32bef4b4495a9444862c95074"}
Oct 30 20:21:44.402600 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:44.402496    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj" event={"ID":"165323da-d482-48d7-8616-0f20cef73c1f","Type":"ContainerStarted","Data":"bc40c8cd9de524b5ab09352bbaf916002c21e5b52b70de53d036a2248215d221"}
Oct 30 20:21:44.422956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:44.422913    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj" podStartSLOduration=1.422897696 podStartE2EDuration="1.422897696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:44.420822315 +0000 UTC m=+1037.324242341" watchObservedRunningTime="2026-10-30 20:21:44.422897696 +0000 UTC m=+1037.326317722"
Oct 30 20:21:45.411461 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:45.411427    6058 generic.go:356] "Generic (PLEG): container finished" podID="165323da-d482-48d7-8616-0f20cef73c1f" containerID="19e0784c7584e6bc704b42bc75d1feb524fac7c32bef4b4495a9444862c95074" exitCode=0
Oct 30 20:21:45.411857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:45.411512    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj" event={"ID":"165323da-d482-48d7-8616-0f20cef73c1f","Type":"ContainerDied","Data":"19e0784c7584e6bc704b42bc75d1feb524fac7c32bef4b4495a9444862c95074"}
Oct 30 20:21:46.634904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.634872    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:46.695329 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.695238    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca\") pod \"165323da-d482-48d7-8616-0f20cef73c1f\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") "
Oct 30 20:21:46.695504 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.695403    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8\") pod \"165323da-d482-48d7-8616-0f20cef73c1f\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") "
Oct 30 20:21:46.695504 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.695446    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca\") pod \"165323da-d482-48d7-8616-0f20cef73c1f\" (UID: \"165323da-d482-48d7-8616-0f20cef73c1f\") "
Oct 30 20:21:46.696093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.696062    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca" pod "165323da-d482-48d7-8616-0f20cef73c1f" (UID: "165323da-d482-48d7-8616-0f20cef73c1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:46.699340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.699314    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8" pod "165323da-d482-48d7-8616-0f20cef73c1f" (UID: "165323da-d482-48d7-8616-0f20cef73c1f"). InnerVolumeSpecName "kube-api-access-tp6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:46.703409 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.703386    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca" pod "165323da-d482-48d7-8616-0f20cef73c1f" (UID: "165323da-d482-48d7-8616-0f20cef73c1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:46.797531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.797500    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/165323da-d482-48d7-8616-0f20cef73c1f-kube-api-access-tp6s8\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:46.797531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.797527    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:46.797531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:46.797538    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165323da-d482-48d7-8616-0f20cef73c1f-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:47.427563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.427537    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"
Oct 30 20:21:47.427730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.427571    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj" event={"ID":"165323da-d482-48d7-8616-0f20cef73c1f","Type":"ContainerDied","Data":"bc40c8cd9de524b5ab09352bbaf916002c21e5b52b70de53d036a2248215d221"}
Oct 30 20:21:47.427730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.427620    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc40c8cd9de524b5ab09352bbaf916002c21e5b52b70de53d036a2248215d221"
Oct 30 20:21:47.789320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.789281    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"]
Oct 30 20:21:47.789919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.789900    6058 cpu_manager.go:414] "containerMap: removing container" podUID="165323da-d482-48d7-8616-0f20cef73c1f" containerName="revoke-keys"
Oct 30 20:21:47.789919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.789919    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165323da-d482-48d7-8616-0f20cef73c1f" containerName="revoke-keys"
Oct 30 20:21:47.790064 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.790043    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="165323da-d482-48d7-8616-0f20cef73c1f" containerName="revoke-keys"
Oct 30 20:21:47.793270 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.793246    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:47.802947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.802922    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"]
Oct 30 20:21:47.805755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.805718    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:47.805755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.805724    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:47.805884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.805771    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:47.805884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.805775    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:47.912689 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.912655    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:47.912857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.912695    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrkz\" (UniqueName: \"kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:47.912857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:47.912801    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.014085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.014046    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.014085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.014088    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrkz\" (UniqueName: \"kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.014278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.014209    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.014821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.014799    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.016354 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.016332    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.027166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.027143    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrkz\" (UniqueName: \"kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz\") pod \"maas-api-revoke-sub-8c14be32-zvkgs\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.112917 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.112891    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:48.262339 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.262295    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:21:48.411299 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.411269    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"]
Oct 30 20:21:48.412231 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:48.412203    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c228f1_61b6_453c_bfe9_8e642629445a.slice/crio-da80b28ef15561eb4685bc957b40520a235da637fc8aff4796c3d69bbe9fe4f5 WatchSource:0}: Error finding container da80b28ef15561eb4685bc957b40520a235da637fc8aff4796c3d69bbe9fe4f5: Status 404 returned error can't find the container with id da80b28ef15561eb4685bc957b40520a235da637fc8aff4796c3d69bbe9fe4f5
Oct 30 20:21:48.440937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:48.440900    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs" event={"ID":"a0c228f1-61b6-453c-bfe9-8e642629445a","Type":"ContainerStarted","Data":"da80b28ef15561eb4685bc957b40520a235da637fc8aff4796c3d69bbe9fe4f5"}
Oct 30 20:21:49.479740 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:49.479704    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs" event={"ID":"a0c228f1-61b6-453c-bfe9-8e642629445a","Type":"ContainerStarted","Data":"bf307cceb7ee8842ca12d20ba0fb5156135ace5f632569c4edf9d70cd8fde5c6"}
Oct 30 20:21:49.525129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:49.521966    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs" podStartSLOduration=2.5219520749999997 podStartE2EDuration="2.521952075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:49.515557243 +0000 UTC m=+1042.418977283" watchObservedRunningTime="2026-10-30 20:21:49.521952075 +0000 UTC m=+1042.425372100"
Oct 30 20:21:49.888029 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:21:49.887311    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:21:50.526031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:50.525648    6058 generic.go:356] "Generic (PLEG): container finished" podID="a0c228f1-61b6-453c-bfe9-8e642629445a" containerID="bf307cceb7ee8842ca12d20ba0fb5156135ace5f632569c4edf9d70cd8fde5c6" exitCode=0
Oct 30 20:21:50.526031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:50.525881    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs" event={"ID":"a0c228f1-61b6-453c-bfe9-8e642629445a","Type":"ContainerDied","Data":"bf307cceb7ee8842ca12d20ba0fb5156135ace5f632569c4edf9d70cd8fde5c6"}
Oct 30 20:21:50.927030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:50.922284    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:50.946538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:50.946494    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.082149 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.082104    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.110160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.110112    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.195918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.195122    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.226348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.225249    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.392669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.392622    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.641256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.641205    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:21:51.771873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.771825    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.875192 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.875137    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.924036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.923085    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.924036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.923866    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:21:51.950271 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.950227    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:51.951965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:51.951941    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:21:52.224088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.223617    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:52.236028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.225425    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:52.236028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.227167    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:21:52.236028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.227229    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:21:52.455193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.250698    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:21:52.455193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.251568    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:21:52.455193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.400616    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:21:52.455193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.400926    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:21:52.675352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.675291    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:52.678307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.678267    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:21:52.772410 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.772353    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:52.847135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.819543    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:21:52.961078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:52.959549    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:53.096026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.095543    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:53.286085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.248181    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:53.290319 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.290276    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:21:53.323516 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.323458    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:53.594117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.521033    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:21:53.594117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.521072    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 20:21:53.701056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.699773    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:53.702517 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.702386    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 30 20:21:53.771031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.723527    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:21:53.922181 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:53.807131    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:54.087795 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.087756    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca\") pod \"a0c228f1-61b6-453c-bfe9-8e642629445a\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") "
Oct 30 20:21:54.088104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.088078    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca\") pod \"a0c228f1-61b6-453c-bfe9-8e642629445a\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") "
Oct 30 20:21:54.106950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.106917    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz\" (UniqueName: \"kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz\") pod \"a0c228f1-61b6-453c-bfe9-8e642629445a\" (UID: \"a0c228f1-61b6-453c-bfe9-8e642629445a\") "
Oct 30 20:21:54.118134 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.118073    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:54.424035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.170878    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:54.424035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.295587    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:54.424035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.296311    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 20:21:54.424035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.303235    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca" pod "a0c228f1-61b6-453c-bfe9-8e642629445a" (UID: "a0c228f1-61b6-453c-bfe9-8e642629445a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:54.424035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.304346    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz" pod "a0c228f1-61b6-453c-bfe9-8e642629445a" (UID: "a0c228f1-61b6-453c-bfe9-8e642629445a"). InnerVolumeSpecName "kube-api-access-hzrkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:54.424035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.304855    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:21:54.464283 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.462757    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.464283 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.462794    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrkz\" (UniqueName: \"kubernetes.io/projected/a0c228f1-61b6-453c-bfe9-8e642629445a-kube-api-access-hzrkz\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.489091 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:21:54.489057    6058 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.166s"
Oct 30 20:21:54.578333 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.578231    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:54.578842 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.578811    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:21:54.687029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.678396    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca" pod "a0c228f1-61b6-453c-bfe9-8e642629445a" (UID: "a0c228f1-61b6-453c-bfe9-8e642629445a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:54.693038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.691474    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c228f1-61b6-453c-bfe9-8e642629445a-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.835776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.835693    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"
Oct 30 20:21:54.884884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.833648    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs" event={"ID":"a0c228f1-61b6-453c-bfe9-8e642629445a","Type":"ContainerDied","Data":"da80b28ef15561eb4685bc957b40520a235da637fc8aff4796c3d69bbe9fe4f5"}
Oct 30 20:21:54.899161 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.899126    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da80b28ef15561eb4685bc957b40520a235da637fc8aff4796c3d69bbe9fe4f5"
Oct 30 20:21:55.028029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.983107    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:55.028029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:54.983383    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:21:55.341290 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.341252    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:21:55.367394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.366852    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"]
Oct 30 20:21:55.415344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.410598    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a0c228f1-61b6-453c-bfe9-8e642629445a" containerName="revoke-keys"
Oct 30 20:21:55.415344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.410631    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c228f1-61b6-453c-bfe9-8e642629445a" containerName="revoke-keys"
Oct 30 20:21:55.415344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.410812    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c228f1-61b6-453c-bfe9-8e642629445a" containerName="revoke-keys"
Oct 30 20:21:55.415344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.412384    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:55.415344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.412970    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:55.415344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.415202    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:21:55.420085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.417392    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:55.425556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.425537    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"]
Oct 30 20:21:55.425993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.425744    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.439800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.439782    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:55.440094 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.440080    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:55.440247 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.440236    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:55.446915 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.446775    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:21:55.447788 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.447770    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:21:55.472026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.469474    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.472026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.469641    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.472026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.469743    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ktz\" (UniqueName: \"kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.542200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.542147    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:21:55.573124 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.570993    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.573124 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.571143    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.573124 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.571212    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ktz\" (UniqueName: \"kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.573124 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.573068    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.613108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.599265    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:21:55.613108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.600394    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:21:55.720711 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.720677    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ktz\" (UniqueName: \"kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.724159 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.724122    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.726206 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.726178    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"]
Oct 30 20:21:55.734071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.734048    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:21:55.734258 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.734247    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:21:55.734406 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.734395    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:21:55.734503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.734493    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"]
Oct 30 20:21:55.734618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.734608    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:21:55.735224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.735201    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:21:55.735505 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.735485    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.788850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.788814    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:55.810184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.810154    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd6wf\" (UniqueName: \"kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.810307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.810202    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.810307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.810269    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.911874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.911842    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd6wf\" (UniqueName: \"kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.912315 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.911893    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.912315 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.911951    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.912728 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.912701    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.914979 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.914950    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:55.931331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:55.931186    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd6wf\" (UniqueName: \"kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf\") pod \"maas-api-revoke-sub-86442197-tk6jt\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:56.049720 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.049690    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:56.101975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.101951    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"]
Oct 30 20:21:56.338343 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:21:56.338311    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014962b9_b1cd_4f86_b667_bdd3f5ca24e6.slice/crio-e2995994c996d4d733ad227005a23463e6154cc6081195650b8f1325b7b02c7d WatchSource:0}: Error finding container e2995994c996d4d733ad227005a23463e6154cc6081195650b8f1325b7b02c7d: Status 404 returned error can't find the container with id e2995994c996d4d733ad227005a23463e6154cc6081195650b8f1325b7b02c7d
Oct 30 20:21:56.338727 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.338704    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"]
Oct 30 20:21:56.862047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.861992    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt" event={"ID":"014962b9-b1cd-4f86-b667-bdd3f5ca24e6","Type":"ContainerStarted","Data":"e2995994c996d4d733ad227005a23463e6154cc6081195650b8f1325b7b02c7d"}
Oct 30 20:21:56.864075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.864038    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9" event={"ID":"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf","Type":"ContainerStarted","Data":"f286e0cfdbf4a92168d63453976a3c2806b9d15455e13185683ea8f9c147d3a8"}
Oct 30 20:21:56.864217 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.864100    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9" event={"ID":"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf","Type":"ContainerStarted","Data":"a56a55b617623fc2affd9e3d3f7f8edead7557e2b7deb312070047ff0b767853"}
Oct 30 20:21:56.884351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:56.884305    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9" podStartSLOduration=1.884290289 podStartE2EDuration="1.884290289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:56.882717088 +0000 UTC m=+1049.786137114" watchObservedRunningTime="2026-10-30 20:21:56.884290289 +0000 UTC m=+1049.787710316"
Oct 30 20:21:57.874733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:57.874695    6058 generic.go:356] "Generic (PLEG): container finished" podID="014962b9-b1cd-4f86-b667-bdd3f5ca24e6" containerID="36e79f688399bf617588cff195d7852c15cf344599ae16afa6cbe7b30f23accf" exitCode=0
Oct 30 20:21:57.875263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:57.874774    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt" event={"ID":"014962b9-b1cd-4f86-b667-bdd3f5ca24e6","Type":"ContainerDied","Data":"36e79f688399bf617588cff195d7852c15cf344599ae16afa6cbe7b30f23accf"}
Oct 30 20:21:57.876679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:57.876651    6058 generic.go:356] "Generic (PLEG): container finished" podID="60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" containerID="f286e0cfdbf4a92168d63453976a3c2806b9d15455e13185683ea8f9c147d3a8" exitCode=0
Oct 30 20:21:57.876825 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:57.876729    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9" event={"ID":"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf","Type":"ContainerDied","Data":"f286e0cfdbf4a92168d63453976a3c2806b9d15455e13185683ea8f9c147d3a8"}
Oct 30 20:21:59.077330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.077304    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:59.080670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.080647    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:59.152283 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152207    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca\") pod \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") "
Oct 30 20:21:59.152449 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152298    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf\" (UniqueName: \"kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf\") pod \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") "
Oct 30 20:21:59.152449 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152357    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz\" (UniqueName: \"kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz\") pod \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") "
Oct 30 20:21:59.152579 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152456    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca\") pod \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\" (UID: \"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf\") "
Oct 30 20:21:59.153637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152759    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca\") pod \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") "
Oct 30 20:21:59.153637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152888    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca\") pod \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\" (UID: \"014962b9-b1cd-4f86-b667-bdd3f5ca24e6\") "
Oct 30 20:21:59.153637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.152920    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca" pod "60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" (UID: "60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:59.153637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.153234    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca" pod "014962b9-b1cd-4f86-b667-bdd3f5ca24e6" (UID: "014962b9-b1cd-4f86-b667-bdd3f5ca24e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:59.153637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.153589    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:59.153637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.153615    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:59.157935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.157903    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca" pod "60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" (UID: "60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:59.158070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.157951    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf" pod "014962b9-b1cd-4f86-b667-bdd3f5ca24e6" (UID: "014962b9-b1cd-4f86-b667-bdd3f5ca24e6"). InnerVolumeSpecName "kube-api-access-kd6wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:59.158142 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.158075    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz" pod "60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" (UID: "60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf"). InnerVolumeSpecName "kube-api-access-l7ktz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:59.158457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.158434    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca" pod "014962b9-b1cd-4f86-b667-bdd3f5ca24e6" (UID: "014962b9-b1cd-4f86-b667-bdd3f5ca24e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:59.255236 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.255206    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:59.255340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.255240    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:59.255340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.255256    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd6wf\" (UniqueName: \"kubernetes.io/projected/014962b9-b1cd-4f86-b667-bdd3f5ca24e6-kube-api-access-kd6wf\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:59.255340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.255272    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ktz\" (UniqueName: \"kubernetes.io/projected/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf-kube-api-access-l7ktz\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:59.630657 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.630619    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"]
Oct 30 20:21:59.631270 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.631250    6058 cpu_manager.go:414] "containerMap: removing container" podUID="60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" containerName="revoke-keys"
Oct 30 20:21:59.631270 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.631268    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" containerName="revoke-keys"
Oct 30 20:21:59.631457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.631318    6058 cpu_manager.go:414] "containerMap: removing container" podUID="014962b9-b1cd-4f86-b667-bdd3f5ca24e6" containerName="revoke-keys"
Oct 30 20:21:59.631457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.631323    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014962b9-b1cd-4f86-b667-bdd3f5ca24e6" containerName="revoke-keys"
Oct 30 20:21:59.631457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.631403    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="014962b9-b1cd-4f86-b667-bdd3f5ca24e6" containerName="revoke-keys"
Oct 30 20:21:59.631457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.631412    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" containerName="revoke-keys"
Oct 30 20:21:59.635130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.635108    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"]
Oct 30 20:21:59.635288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.635259    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.647903 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.647782    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"]
Oct 30 20:21:59.660687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.660657    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9228q\" (UniqueName: \"kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.660830 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.660817    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.660932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.660868    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.761795 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.761744    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.761990 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.761812    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.762101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.761989    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9228q\" (UniqueName: \"kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.762584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.762555    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.764374 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.764351    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.775113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.775093    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9228q\" (UniqueName: \"kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q\") pod \"maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:21:59.894768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.894689    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"
Oct 30 20:21:59.894929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.894686    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt" event={"ID":"014962b9-b1cd-4f86-b667-bdd3f5ca24e6","Type":"ContainerDied","Data":"e2995994c996d4d733ad227005a23463e6154cc6081195650b8f1325b7b02c7d"}
Oct 30 20:21:59.894929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.894812    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2995994c996d4d733ad227005a23463e6154cc6081195650b8f1325b7b02c7d"
Oct 30 20:21:59.896170 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.896140    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"
Oct 30 20:21:59.896170 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.896159    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9" event={"ID":"60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf","Type":"ContainerDied","Data":"a56a55b617623fc2affd9e3d3f7f8edead7557e2b7deb312070047ff0b767853"}
Oct 30 20:21:59.896328 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.896193    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a56a55b617623fc2affd9e3d3f7f8edead7557e2b7deb312070047ff0b767853"
Oct 30 20:21:59.970307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:21:59.970281    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:22:00.255864 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:00.255828    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda36db1d_7117_41a6_b347_c66f7eec4921.slice/crio-27afda9b1d8ddf7a6bff563d49a548cba0aa9c80149a07f9e61a6d2306c72a0b WatchSource:0}: Error finding container 27afda9b1d8ddf7a6bff563d49a548cba0aa9c80149a07f9e61a6d2306c72a0b: Status 404 returned error can't find the container with id 27afda9b1d8ddf7a6bff563d49a548cba0aa9c80149a07f9e61a6d2306c72a0b
Oct 30 20:22:00.256292 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:00.256276    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"]
Oct 30 20:22:00.906679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:00.906636    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh" event={"ID":"da36db1d-7117-41a6-b347-c66f7eec4921","Type":"ContainerStarted","Data":"8d37b450bfd5d26c096bdb159ca1523aa276c2758df4d866d016980833e4c165"}
Oct 30 20:22:00.906846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:00.906744    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh" event={"ID":"da36db1d-7117-41a6-b347-c66f7eec4921","Type":"ContainerStarted","Data":"27afda9b1d8ddf7a6bff563d49a548cba0aa9c80149a07f9e61a6d2306c72a0b"}
Oct 30 20:22:00.929394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:00.929343    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh" podStartSLOduration=1.929330533 podStartE2EDuration="1.929330533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:00.925362355 +0000 UTC m=+1053.828782395" watchObservedRunningTime="2026-10-30 20:22:00.929330533 +0000 UTC m=+1053.832750559"
Oct 30 20:22:01.919828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:01.919789    6058 generic.go:356] "Generic (PLEG): container finished" podID="da36db1d-7117-41a6-b347-c66f7eec4921" containerID="8d37b450bfd5d26c096bdb159ca1523aa276c2758df4d866d016980833e4c165" exitCode=0
Oct 30 20:22:01.920465 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:01.920432    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh" event={"ID":"da36db1d-7117-41a6-b347-c66f7eec4921","Type":"ContainerDied","Data":"8d37b450bfd5d26c096bdb159ca1523aa276c2758df4d866d016980833e4c165"}
Oct 30 20:22:03.088403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.088373    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:22:03.110049 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.109969    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca\") pod \"da36db1d-7117-41a6-b347-c66f7eec4921\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") "
Oct 30 20:22:03.110273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.110245    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca\") pod \"da36db1d-7117-41a6-b347-c66f7eec4921\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") "
Oct 30 20:22:03.111281 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.110298    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q\" (UniqueName: \"kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q\") pod \"da36db1d-7117-41a6-b347-c66f7eec4921\" (UID: \"da36db1d-7117-41a6-b347-c66f7eec4921\") "
Oct 30 20:22:03.111281 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.111107    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca" pod "da36db1d-7117-41a6-b347-c66f7eec4921" (UID: "da36db1d-7117-41a6-b347-c66f7eec4921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:03.112848 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.112805    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca" pod "da36db1d-7117-41a6-b347-c66f7eec4921" (UID: "da36db1d-7117-41a6-b347-c66f7eec4921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:03.113223 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.113183    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q" pod "da36db1d-7117-41a6-b347-c66f7eec4921" (UID: "da36db1d-7117-41a6-b347-c66f7eec4921"). InnerVolumeSpecName "kube-api-access-9228q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:03.211617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.211527    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:03.211617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.211556    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9228q\" (UniqueName: \"kubernetes.io/projected/da36db1d-7117-41a6-b347-c66f7eec4921-kube-api-access-9228q\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:03.211617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.211566    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36db1d-7117-41a6-b347-c66f7eec4921-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:03.946499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.946466    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"
Oct 30 20:22:03.946671 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.946508    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh" event={"ID":"da36db1d-7117-41a6-b347-c66f7eec4921","Type":"ContainerDied","Data":"27afda9b1d8ddf7a6bff563d49a548cba0aa9c80149a07f9e61a6d2306c72a0b"}
Oct 30 20:22:03.946671 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:03.946566    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27afda9b1d8ddf7a6bff563d49a548cba0aa9c80149a07f9e61a6d2306c72a0b"
Oct 30 20:22:04.461558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:04.461519    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"]
Oct 30 20:22:04.491681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:04.491644    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"]
Oct 30 20:22:05.577387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.577354    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"]
Oct 30 20:22:05.577954 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.577939    6058 cpu_manager.go:414] "containerMap: removing container" podUID="da36db1d-7117-41a6-b347-c66f7eec4921" containerName="revoke-keys"
Oct 30 20:22:05.577999 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.577954    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da36db1d-7117-41a6-b347-c66f7eec4921" containerName="revoke-keys"
Oct 30 20:22:05.578078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.578068    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="da36db1d-7117-41a6-b347-c66f7eec4921" containerName="revoke-keys"
Oct 30 20:22:05.582093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.582069    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.588231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.588206    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:05.588589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.588570    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:05.588742 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.588706    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:22:05.589258 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.589182    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:05.593277 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.592697    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"]
Oct 30 20:22:05.649345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.649311    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.649494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.649434    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.649596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.649576    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp5w\" (UniqueName: \"kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.750877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.750842    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp5w\" (UniqueName: \"kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.751084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.750998    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.751173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.751107    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.751682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.751652    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.753479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.753454    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.771559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.771533    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"]
Oct 30 20:22:05.771814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.771785    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp5w\" (UniqueName: \"kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w\") pod \"maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:05.907876 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:05.907844    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:06.242384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:06.242350    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"]
Oct 30 20:22:06.976940 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:06.976909    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd" event={"ID":"87374534-095a-4e79-af77-aa9e0717fe49","Type":"ContainerStarted","Data":"0a7808c6c2f09f9b6fb49a2494a6b3fab63f49a8618d234a79acd5f3406707d6"}
Oct 30 20:22:06.977358 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:06.976972    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd" event={"ID":"87374534-095a-4e79-af77-aa9e0717fe49","Type":"ContainerStarted","Data":"721914271d9bf3b86ab625dc102e4ec5c2142d0f892475b989d32ad4dcca7f91"}
Oct 30 20:22:07.001893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:07.001840    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd" podStartSLOduration=2.001827181 podStartE2EDuration="2.001827181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:06.996903763 +0000 UTC m=+1059.900323788" watchObservedRunningTime="2026-10-30 20:22:07.001827181 +0000 UTC m=+1059.905247206"
Oct 30 20:22:07.991399 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:07.991352    6058 generic.go:356] "Generic (PLEG): container finished" podID="87374534-095a-4e79-af77-aa9e0717fe49" containerID="0a7808c6c2f09f9b6fb49a2494a6b3fab63f49a8618d234a79acd5f3406707d6" exitCode=0
Oct 30 20:22:07.991885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:07.991596    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd" event={"ID":"87374534-095a-4e79-af77-aa9e0717fe49","Type":"ContainerDied","Data":"0a7808c6c2f09f9b6fb49a2494a6b3fab63f49a8618d234a79acd5f3406707d6"}
Oct 30 20:22:09.245732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.245705    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:09.326167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.326088    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca\") pod \"87374534-095a-4e79-af77-aa9e0717fe49\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") "
Oct 30 20:22:09.326167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.326145    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w\" (UniqueName: \"kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w\") pod \"87374534-095a-4e79-af77-aa9e0717fe49\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") "
Oct 30 20:22:09.326371 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.326346    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca\") pod \"87374534-095a-4e79-af77-aa9e0717fe49\" (UID: \"87374534-095a-4e79-af77-aa9e0717fe49\") "
Oct 30 20:22:09.326528 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.326500    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca" pod "87374534-095a-4e79-af77-aa9e0717fe49" (UID: "87374534-095a-4e79-af77-aa9e0717fe49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:09.327202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.327177    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:09.328595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.328572    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca" pod "87374534-095a-4e79-af77-aa9e0717fe49" (UID: "87374534-095a-4e79-af77-aa9e0717fe49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:09.328716 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.328643    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w" pod "87374534-095a-4e79-af77-aa9e0717fe49" (UID: "87374534-095a-4e79-af77-aa9e0717fe49"). InnerVolumeSpecName "kube-api-access-9pp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:09.428905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.428869    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pp5w\" (UniqueName: \"kubernetes.io/projected/87374534-095a-4e79-af77-aa9e0717fe49-kube-api-access-9pp5w\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:09.428905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.428902    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87374534-095a-4e79-af77-aa9e0717fe49-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:09.824167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.824134    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"]
Oct 30 20:22:09.879226 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.879175    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"]
Oct 30 20:22:09.880401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.879870    6058 cpu_manager.go:414] "containerMap: removing container" podUID="87374534-095a-4e79-af77-aa9e0717fe49" containerName="revoke-keys"
Oct 30 20:22:09.880401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.879893    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87374534-095a-4e79-af77-aa9e0717fe49" containerName="revoke-keys"
Oct 30 20:22:09.880401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.880125    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="87374534-095a-4e79-af77-aa9e0717fe49" containerName="revoke-keys"
Oct 30 20:22:09.890629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.887901    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:09.915044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:09.905038    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"]
Oct 30 20:22:10.029689 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.029661    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"
Oct 30 20:22:10.029880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.029710    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd" event={"ID":"87374534-095a-4e79-af77-aa9e0717fe49","Type":"ContainerDied","Data":"721914271d9bf3b86ab625dc102e4ec5c2142d0f892475b989d32ad4dcca7f91"}
Oct 30 20:22:10.029880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.029776    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721914271d9bf3b86ab625dc102e4ec5c2142d0f892475b989d32ad4dcca7f91"
Oct 30 20:22:10.038084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038048    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038106    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038163    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038203    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038231    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038332    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038355    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038437    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.038656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.038490    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphbl\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.139902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.139871    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphbl\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.139935    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.139961    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.139995    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140067    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140088    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140165    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140182    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140281    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140473    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140524    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140644 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140581    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.140780 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.140750    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.141334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.141306    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.142321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.142299    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.142552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.142534    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.153045 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.152986    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphbl\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.153158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.153060    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token\") pod \"e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") " pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.217575 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.217529    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:10.341772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.331383    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"]
Oct 30 20:22:10.685670 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:10.685628    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3ac4a9_ff39_449e_8cc9_a58c5ba8d4fa.slice/crio-24b03769103f90af776765dadbb033aea0f42fb33b81e1e4a50ec6236f866ab2 WatchSource:0}: Error finding container 24b03769103f90af776765dadbb033aea0f42fb33b81e1e4a50ec6236f866ab2: Status 404 returned error can't find the container with id 24b03769103f90af776765dadbb033aea0f42fb33b81e1e4a50ec6236f866ab2
Oct 30 20:22:10.688163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.688136    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"]
Oct 30 20:22:10.688401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.688364    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:22:10.688497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.688448    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:22:10.688562 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.688504    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:22:10.772876 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:10.772811    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerName="payload-processing" containerID="cri-o://1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6" gracePeriod=25
Oct 30 20:22:11.039612 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.039523    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" event={"ID":"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa","Type":"ContainerStarted","Data":"62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664"}
Oct 30 20:22:11.039612 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.039583    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" event={"ID":"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa","Type":"ContainerStarted","Data":"24b03769103f90af776765dadbb033aea0f42fb33b81e1e4a50ec6236f866ab2"}
Oct 30 20:22:11.045846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.045816    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"]
Oct 30 20:22:11.051341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.051318    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.056860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.056834    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:11.057041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.056949    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:11.057113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.057071    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:11.057232 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.057213    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:22:11.063572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.063279    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"]
Oct 30 20:22:11.069969 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.069104    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.093190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.093140    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"]
Oct 30 20:22:11.100937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.099443    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"]
Oct 30 20:22:11.105416 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.102770    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podStartSLOduration=2.102757901 podStartE2EDuration="2.102757901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:11.074178402 +0000 UTC m=+1063.977598429" watchObservedRunningTime="2026-10-30 20:22:11.102757901 +0000 UTC m=+1064.006177926"
Oct 30 20:22:11.194029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.193095    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vln9n\" (UniqueName: \"kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.194029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.193213    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.197030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.194940    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.197030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.195144    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.197030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.195268    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.197030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.195334    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9j9\" (UniqueName: \"kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.218422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.218296    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:11.246317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.246271    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:11.246493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.246368    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.303304    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.303422    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.303534    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.303639    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.303696    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9j9\" (UniqueName: \"kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.303779    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vln9n\" (UniqueName: \"kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.309610    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.310516    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.318472    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.324497    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vln9n\" (UniqueName: \"kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.328362    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7fc1f8-l6xhd\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:11.364082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.352597    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9j9\" (UniqueName: \"kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9\") pod \"maas-api-revoke-sub-c89c471f-6rhpc\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.384302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.384267    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:11.408108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:11.406682    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:12.222033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.219360    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:12.222033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.219420    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:12.282038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.278560    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"]
Oct 30 20:22:12.375448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.375421    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.378752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.378722    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"]
Oct 30 20:22:12.486216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.484171    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.486216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.484247    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.486216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.484303    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbtx\" (UniqueName: \"kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.601044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.593097    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.601044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.593183    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.601044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.593244    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbtx\" (UniqueName: \"kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.661563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.659103    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbtx\" (UniqueName: \"kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.686045 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.681525    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.743108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.728733    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b522314-n2fg2\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:12.795058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:12.787482    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:13.033997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.022777    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:22:13.173030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.172050    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"]
Oct 30 20:22:13.375028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.368190    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:13.375028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.368303    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:13.879664 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.852069    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"]
Oct 30 20:22:13.879664 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:13.852650    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42fc25c_a8fe_402e_841b_abd0014266a2.slice/crio-4dbdd7cb80493c97099563a5997f19b359e6814ce973095ab5e6a484d73cb7f1 WatchSource:0}: Error finding container 4dbdd7cb80493c97099563a5997f19b359e6814ce973095ab5e6a484d73cb7f1: Status 404 returned error can't find the container with id 4dbdd7cb80493c97099563a5997f19b359e6814ce973095ab5e6a484d73cb7f1
Oct 30 20:22:14.080684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.957164    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 20:22:14.080684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.966268    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:14.080684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:13.986030    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"]
Oct 30 20:22:14.080684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.029567    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:14.080684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.030723    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:22:14.286038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.285329    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:14.394624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.388713    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:14.394624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.388775    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:14.765466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.434103    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:22:14.765466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.603281    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:14.765466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.604516    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:22:14.765466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.763088    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.831931    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing" containerID="cri-o://e4bc98ed92643d0df4bab799b255d7453b6ca550191d87ca2962bbb58f7318be" gracePeriod=25
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.834127    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" containerID="cri-o://39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e" gracePeriod=45
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.961570    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.966136    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.966958    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.973331    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.976732    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"]
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.993438    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": read tcp 10.133.0.2:49812->10.133.0.48:8000: read: connection reset by peer"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:14.993540    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.126754    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:22:15.412691 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.330933    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 20:22:15.713818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.448103    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:22:15.713818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.449894    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:15.713818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.449937    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:15.713818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.599934    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 20:22:15.713818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.620321    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 20:22:15.972413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:15.870331    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:22:16.231817 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.231725    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:22:16.232104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.232083    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:22:16.232255 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.232231    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc" event={"ID":"a42fc25c-a8fe-402e-841b-abd0014266a2","Type":"ContainerStarted","Data":"4dbdd7cb80493c97099563a5997f19b359e6814ce973095ab5e6a484d73cb7f1"}
Oct 30 20:22:16.232480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.232464    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:22:16.232638 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.232624    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:22:16.358992 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.358924    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:16.373321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.373273    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:16.399041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.382486    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:16.399041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.398579    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:16.561026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.557411    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:16.593302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.593168    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:16.596591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.596528    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 20:22:16.673245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.655735    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:16.857085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.831804    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 20:22:16.898424 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.898372    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:16.971031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.946195    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m8gtk" podUID="6a19f9af-b5c9-4122-a0e4-20f215eef257" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 20:22:16.973607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:16.971808    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:22:17.145051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.135640    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:17.374401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.373325    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:17.722084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.419804    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 20:22:17.722084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.420326    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:22:17.835422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.612449    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:17.835422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.748092    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:17.835422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.659915    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 20:22:17.915673 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.908103    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:17.934094 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.916461    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:17.950803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.950763    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd" event={"ID":"62e0637e-89e4-4f94-90e0-12524247f76c","Type":"ContainerStarted","Data":"7e13c11638df526182311aef1cdb1994e4b5c30f6de70ad9cd6082a5065cf915"}
Oct 30 20:22:18.001234 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:17.982160    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:18.129497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.129467    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:22:18.130212 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.129919    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:22:18.135308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.133796    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:22:18.135308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.133849    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:22:18.135308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.133877    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:22:18.135308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.133906    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:22:18.135744 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:18.135579    6058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8cc6c8_8f40_40af_bc28_b3699dab3c88.slice/crio-conmon-39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8cc6c8_8f40_40af_bc28_b3699dab3c88.slice/crio-39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:22:18.137522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.137502    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:22:18.144654 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:18.144623    6058 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8cc6c8_8f40_40af_bc28_b3699dab3c88.slice/crio-conmon-39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:22:18.219069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.218525    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:18.219069 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.218578    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:18.346902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.346866    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd" event={"ID":"62e0637e-89e4-4f94-90e0-12524247f76c","Type":"ContainerStarted","Data":"ad0387d7c9c899f92458fd4d132c0b0a97181e83cdf36e14d35a83e4f32b6ff6"}
Oct 30 20:22:18.395324 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.395284    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc" event={"ID":"a42fc25c-a8fe-402e-841b-abd0014266a2","Type":"ContainerStarted","Data":"b5a761b894cfd362a5075430b935799d494c23d78587faa89aa8ddd2b9f88da3"}
Oct 30 20:22:18.399291 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.399260    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2" event={"ID":"4b29394f-045e-495b-ab74-adaadec9f485","Type":"ContainerStarted","Data":"0feba03ea538d65424e9d7b952cc1f57b836727e9ed0fcdaf0659bff9aeb6391"}
Oct 30 20:22:18.403151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.403120    6058 generic.go:356] "Generic (PLEG): container finished" podID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerID="39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e" exitCode=0
Oct 30 20:22:18.404293 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.404266    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" event={"ID":"6d8cc6c8-8f40-40af-bc28-b3699dab3c88","Type":"ContainerDied","Data":"39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e"}
Oct 30 20:22:18.447476 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.447268    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc" podStartSLOduration=7.447254392 podStartE2EDuration="7.447254392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:18.443351558 +0000 UTC m=+1071.346771584" watchObservedRunningTime="2026-10-30 20:22:18.447254392 +0000 UTC m=+1071.350674418"
Oct 30 20:22:18.845847 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.845814    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:22:18.889049 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.886294    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd" podStartSLOduration=7.886273385 podStartE2EDuration="7.886273385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:18.476277427 +0000 UTC m=+1071.379697453" watchObservedRunningTime="2026-10-30 20:22:18.886273385 +0000 UTC m=+1071.789693412"
Oct 30 20:22:18.943863 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.943806    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir\") pod \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") "
Oct 30 20:22:18.944181 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.943832    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir" pod "6d8cc6c8-8f40-40af-bc28-b3699dab3c88" (UID: "6d8cc6c8-8f40-40af-bc28-b3699dab3c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:18.944448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.944426    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm\") pod \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") "
Oct 30 20:22:18.944654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.944623    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs\") pod \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") "
Oct 30 20:22:18.944869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.944852    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s\" (UniqueName: \"kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s\") pod \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") "
Oct 30 20:22:18.945078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.945059    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home\") pod \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") "
Oct 30 20:22:18.945384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.945338    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home" pod "6d8cc6c8-8f40-40af-bc28-b3699dab3c88" (UID: "6d8cc6c8-8f40-40af-bc28-b3699dab3c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:18.945384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.945363    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache\") pod \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\" (UID: \"6d8cc6c8-8f40-40af-bc28-b3699dab3c88\") "
Oct 30 20:22:18.945725 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.945698    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache" pod "6d8cc6c8-8f40-40af-bc28-b3699dab3c88" (UID: "6d8cc6c8-8f40-40af-bc28-b3699dab3c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:18.946591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.946567    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm" pod "6d8cc6c8-8f40-40af-bc28-b3699dab3c88" (UID: "6d8cc6c8-8f40-40af-bc28-b3699dab3c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:18.946904 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.946880    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.946982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.946925    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.946982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.946952    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.946982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.946966    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:18.949253 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.949217    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s" pod "6d8cc6c8-8f40-40af-bc28-b3699dab3c88" (UID: "6d8cc6c8-8f40-40af-bc28-b3699dab3c88"). InnerVolumeSpecName "kube-api-access-svj6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:18.968139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:18.961501    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs" pod "6d8cc6c8-8f40-40af-bc28-b3699dab3c88" (UID: "6d8cc6c8-8f40-40af-bc28-b3699dab3c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:19.047917 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.047883    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svj6s\" (UniqueName: \"kubernetes.io/projected/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-kube-api-access-svj6s\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:19.047917 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.047913    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cc6c8-8f40-40af-bc28-b3699dab3c88-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:19.219855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.219762    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:19.220219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.220188    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:19.416669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.416631    6058 generic.go:356] "Generic (PLEG): container finished" podID="62e0637e-89e4-4f94-90e0-12524247f76c" containerID="ad0387d7c9c899f92458fd4d132c0b0a97181e83cdf36e14d35a83e4f32b6ff6" exitCode=0
Oct 30 20:22:19.417108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.416717    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd" event={"ID":"62e0637e-89e4-4f94-90e0-12524247f76c","Type":"ContainerDied","Data":"ad0387d7c9c899f92458fd4d132c0b0a97181e83cdf36e14d35a83e4f32b6ff6"}
Oct 30 20:22:19.418869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.418850    6058 generic.go:356] "Generic (PLEG): container finished" podID="a42fc25c-a8fe-402e-841b-abd0014266a2" containerID="b5a761b894cfd362a5075430b935799d494c23d78587faa89aa8ddd2b9f88da3" exitCode=0
Oct 30 20:22:19.419033 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.418931    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc" event={"ID":"a42fc25c-a8fe-402e-841b-abd0014266a2","Type":"ContainerDied","Data":"b5a761b894cfd362a5075430b935799d494c23d78587faa89aa8ddd2b9f88da3"}
Oct 30 20:22:19.420531 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.420514    6058 generic.go:356] "Generic (PLEG): container finished" podID="4b29394f-045e-495b-ab74-adaadec9f485" containerID="b3aaa633dd670f61f5b3cee0eb8cd8423ca4da4378f77e7b219feba036fbf57c" exitCode=0
Oct 30 20:22:19.420602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.420566    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2" event={"ID":"4b29394f-045e-495b-ab74-adaadec9f485","Type":"ContainerDied","Data":"b3aaa633dd670f61f5b3cee0eb8cd8423ca4da4378f77e7b219feba036fbf57c"}
Oct 30 20:22:19.421893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.421865    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" event={"ID":"6d8cc6c8-8f40-40af-bc28-b3699dab3c88","Type":"ContainerDied","Data":"2297833b08ee41186210f3d57adef4bfe36e778fc60e05940ee11f3b287f5fdd"}
Oct 30 20:22:19.421978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.421909    6058 scope.go:122] "RemoveContainer" containerID="39d1b60250154e541630c75a2936e5ec509014fb8fff30018351457c9f279c4e"
Oct 30 20:22:19.422093 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.422078    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"
Oct 30 20:22:19.490146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.487158    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main" containerID="cri-o://b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2" gracePeriod=45
Oct 30 20:22:19.511749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.511716    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main" containerID="cri-o://e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc" gracePeriod=45
Oct 30 20:22:19.528131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.528098    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"]
Oct 30 20:22:19.533631 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.533599    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh"]
Oct 30 20:22:19.598029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.591113    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c646da/premium-w0-c646da-kserve-9bcdbf6f8-99bqh" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:19.705049 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:19.698221    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:22:19.902160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.902123    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:22:19.933629 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:19.933604    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:22:20.015022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.014968    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir\") pod \"c1731030-822c-4017-b1e5-62c7a8d013fa\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") "
Oct 30 20:22:20.015214 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015176    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir\") pod \"cf801a67-90b1-408d-a14b-86907900d29d\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") "
Oct 30 20:22:20.015285 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015254    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache\") pod \"c1731030-822c-4017-b1e5-62c7a8d013fa\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") "
Oct 30 20:22:20.015325 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015273    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir" pod "c1731030-822c-4017-b1e5-62c7a8d013fa" (UID: "c1731030-822c-4017-b1e5-62c7a8d013fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.015325 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015315    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r\" (UniqueName: \"kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r\") pod \"c1731030-822c-4017-b1e5-62c7a8d013fa\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") "
Oct 30 20:22:20.015385 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015361    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home\") pod \"cf801a67-90b1-408d-a14b-86907900d29d\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") "
Oct 30 20:22:20.015461 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015447    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv\" (UniqueName: \"kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv\") pod \"cf801a67-90b1-408d-a14b-86907900d29d\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") "
Oct 30 20:22:20.015522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015509    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs\") pod \"c1731030-822c-4017-b1e5-62c7a8d013fa\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") "
Oct 30 20:22:20.015616 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015603    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home\") pod \"c1731030-822c-4017-b1e5-62c7a8d013fa\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") "
Oct 30 20:22:20.015669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015642    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache\") pod \"cf801a67-90b1-408d-a14b-86907900d29d\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") "
Oct 30 20:22:20.015717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015704    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm\") pod \"c1731030-822c-4017-b1e5-62c7a8d013fa\" (UID: \"c1731030-822c-4017-b1e5-62c7a8d013fa\") "
Oct 30 20:22:20.015798 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015784    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm\") pod \"cf801a67-90b1-408d-a14b-86907900d29d\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") "
Oct 30 20:22:20.015976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.015953    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs\") pod \"cf801a67-90b1-408d-a14b-86907900d29d\" (UID: \"cf801a67-90b1-408d-a14b-86907900d29d\") "
Oct 30 20:22:20.016624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016124    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home" pod "c1731030-822c-4017-b1e5-62c7a8d013fa" (UID: "c1731030-822c-4017-b1e5-62c7a8d013fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.016624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016206    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache" pod "c1731030-822c-4017-b1e5-62c7a8d013fa" (UID: "c1731030-822c-4017-b1e5-62c7a8d013fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.016873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016760    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir" pod "cf801a67-90b1-408d-a14b-86907900d29d" (UID: "cf801a67-90b1-408d-a14b-86907900d29d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.016873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016808    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home" pod "cf801a67-90b1-408d-a14b-86907900d29d" (UID: "cf801a67-90b1-408d-a14b-86907900d29d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.016873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016810    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.016873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016835    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.016873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016847    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.017084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.016871    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache" pod "cf801a67-90b1-408d-a14b-86907900d29d" (UID: "cf801a67-90b1-408d-a14b-86907900d29d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.018363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.018343    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r" pod "c1731030-822c-4017-b1e5-62c7a8d013fa" (UID: "c1731030-822c-4017-b1e5-62c7a8d013fa"). InnerVolumeSpecName "kube-api-access-z924r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:20.018430 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.018371    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv" pod "cf801a67-90b1-408d-a14b-86907900d29d" (UID: "cf801a67-90b1-408d-a14b-86907900d29d"). InnerVolumeSpecName "kube-api-access-llsbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:20.019419 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.019384    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm" pod "c1731030-822c-4017-b1e5-62c7a8d013fa" (UID: "c1731030-822c-4017-b1e5-62c7a8d013fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.019771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.019745    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs" pod "c1731030-822c-4017-b1e5-62c7a8d013fa" (UID: "c1731030-822c-4017-b1e5-62c7a8d013fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:20.020696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.020669    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs" pod "cf801a67-90b1-408d-a14b-86907900d29d" (UID: "cf801a67-90b1-408d-a14b-86907900d29d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:20.021203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.021176    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm" pod "cf801a67-90b1-408d-a14b-86907900d29d" (UID: "cf801a67-90b1-408d-a14b-86907900d29d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:20.118412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118381    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf801a67-90b1-408d-a14b-86907900d29d-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118409    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118422    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z924r\" (UniqueName: \"kubernetes.io/projected/c1731030-822c-4017-b1e5-62c7a8d013fa-kube-api-access-z924r\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118434    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118442    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llsbv\" (UniqueName: \"kubernetes.io/projected/cf801a67-90b1-408d-a14b-86907900d29d-kube-api-access-llsbv\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118451    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1731030-822c-4017-b1e5-62c7a8d013fa-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118460    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118469    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1731030-822c-4017-b1e5-62c7a8d013fa-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.118776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.118476    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf801a67-90b1-408d-a14b-86907900d29d-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.218627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.218593    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:20.218814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.218646    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:20.433584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.433503    6058 generic.go:356] "Generic (PLEG): container finished" podID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerID="b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2" exitCode=0
Oct 30 20:22:20.433584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.433548    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"
Oct 30 20:22:20.433937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.433591    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" event={"ID":"c1731030-822c-4017-b1e5-62c7a8d013fa","Type":"ContainerDied","Data":"b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2"}
Oct 30 20:22:20.433937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.433630    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw" event={"ID":"c1731030-822c-4017-b1e5-62c7a8d013fa","Type":"ContainerDied","Data":"53d6c4b0b9928d972167e54da458d43b321402cea8f6bde7e7b68c1ff2b921d5"}
Oct 30 20:22:20.433937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.433652    6058 scope.go:122] "RemoveContainer" containerID="b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2"
Oct 30 20:22:20.435316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.435293    6058 generic.go:356] "Generic (PLEG): container finished" podID="cf801a67-90b1-408d-a14b-86907900d29d" containerID="e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc" exitCode=0
Oct 30 20:22:20.435434 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.435332    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"
Oct 30 20:22:20.435434 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.435361    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" event={"ID":"cf801a67-90b1-408d-a14b-86907900d29d","Type":"ContainerDied","Data":"e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc"}
Oct 30 20:22:20.435434 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.435391    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt" event={"ID":"cf801a67-90b1-408d-a14b-86907900d29d","Type":"ContainerDied","Data":"b66a735b5865907a4c6bcc31dd59532289af8c510f9b9e170d92f0275accd361"}
Oct 30 20:22:20.449317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.449300    6058 scope.go:122] "RemoveContainer" containerID="b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2"
Oct 30 20:22:20.450159 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:20.450131    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2\": container with ID starting with b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2 not found: ID does not exist" containerID="b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2"
Oct 30 20:22:20.450241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.450176    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2"} err="failed to get container status \"b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2\": rpc error: code = NotFound desc = could not find container \"b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2\": container with ID starting with b2e216a85a45646c17b1af155237c1488c260fcdf31782294c96ac4ce29bcfa2 not found: ID does not exist"
Oct 30 20:22:20.450241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.450199    6058 scope.go:122] "RemoveContainer" containerID="e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc"
Oct 30 20:22:20.505166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.505099    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"]
Oct 30 20:22:20.512311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.511130    6058 scope.go:122] "RemoveContainer" containerID="e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc"
Oct 30 20:22:20.512311 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:20.511516    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc\": container with ID starting with e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc not found: ID does not exist" containerID="e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc"
Oct 30 20:22:20.512311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.511553    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc"} err="failed to get container status \"e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc\": rpc error: code = NotFound desc = could not find container \"e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc\": container with ID starting with e9f4706a51ad8b488c69a4f3c33e22fad9a2ef0b6068353349037a745d9149cc not found: ID does not exist"
Oct 30 20:22:20.512311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.511813    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/distinct-2-w6-75e734-kserve-58dcfd4999-ts8fw"]
Oct 30 20:22:20.523635 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.523604    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"]
Oct 30 20:22:20.529064 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.529029    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-75e734/simulator-w6-75e734-kserve-69c99c89c6-5b6dt"]
Oct 30 20:22:20.709284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.709200    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:20.758578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.758468    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:20.765161 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.765138    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:20.828266 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.828221    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca\") pod \"a42fc25c-a8fe-402e-841b-abd0014266a2\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") "
Oct 30 20:22:20.828622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.828599    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx\" (UniqueName: \"kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx\") pod \"4b29394f-045e-495b-ab74-adaadec9f485\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") "
Oct 30 20:22:20.828779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.828760    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca\") pod \"62e0637e-89e4-4f94-90e0-12524247f76c\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") "
Oct 30 20:22:20.828912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.828625    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca" pod "a42fc25c-a8fe-402e-841b-abd0014266a2" (UID: "a42fc25c-a8fe-402e-841b-abd0014266a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:20.829037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.829018    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca\") pod \"4b29394f-045e-495b-ab74-adaadec9f485\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") "
Oct 30 20:22:20.829190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.829172    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca\") pod \"4b29394f-045e-495b-ab74-adaadec9f485\" (UID: \"4b29394f-045e-495b-ab74-adaadec9f485\") "
Oct 30 20:22:20.829315 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.829299    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9\" (UniqueName: \"kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9\") pod \"a42fc25c-a8fe-402e-841b-abd0014266a2\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") "
Oct 30 20:22:20.830655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.830145    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n\" (UniqueName: \"kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n\") pod \"62e0637e-89e4-4f94-90e0-12524247f76c\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") "
Oct 30 20:22:20.830655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.829479    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca" pod "62e0637e-89e4-4f94-90e0-12524247f76c" (UID: "62e0637e-89e4-4f94-90e0-12524247f76c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:20.830655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.830186    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca\") pod \"a42fc25c-a8fe-402e-841b-abd0014266a2\" (UID: \"a42fc25c-a8fe-402e-841b-abd0014266a2\") "
Oct 30 20:22:20.830655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.829952    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca" pod "4b29394f-045e-495b-ab74-adaadec9f485" (UID: "4b29394f-045e-495b-ab74-adaadec9f485"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:20.830655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.830280    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca\") pod \"62e0637e-89e4-4f94-90e0-12524247f76c\" (UID: \"62e0637e-89e4-4f94-90e0-12524247f76c\") "
Oct 30 20:22:20.831073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.830957    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.831073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.830981    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.831212 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.830997    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.831443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.831425    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx" pod "4b29394f-045e-495b-ab74-adaadec9f485" (UID: "4b29394f-045e-495b-ab74-adaadec9f485"). InnerVolumeSpecName "kube-api-access-cvbtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:20.833523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.833495    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca" pod "62e0637e-89e4-4f94-90e0-12524247f76c" (UID: "62e0637e-89e4-4f94-90e0-12524247f76c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:20.834310 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.834289    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9" pod "a42fc25c-a8fe-402e-841b-abd0014266a2" (UID: "a42fc25c-a8fe-402e-841b-abd0014266a2"). InnerVolumeSpecName "kube-api-access-qg9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:20.834426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.834405    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n" pod "62e0637e-89e4-4f94-90e0-12524247f76c" (UID: "62e0637e-89e4-4f94-90e0-12524247f76c"). InnerVolumeSpecName "kube-api-access-vln9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:20.834945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.834926    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca" pod "4b29394f-045e-495b-ab74-adaadec9f485" (UID: "4b29394f-045e-495b-ab74-adaadec9f485"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:20.836520 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.836501    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca" pod "a42fc25c-a8fe-402e-841b-abd0014266a2" (UID: "a42fc25c-a8fe-402e-841b-abd0014266a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:20.847594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.847569    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"]
Oct 30 20:22:20.932854 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.932818    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvbtx\" (UniqueName: \"kubernetes.io/projected/4b29394f-045e-495b-ab74-adaadec9f485-kube-api-access-cvbtx\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.932854 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.932852    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b29394f-045e-495b-ab74-adaadec9f485-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.933418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.932868    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg9j9\" (UniqueName: \"kubernetes.io/projected/a42fc25c-a8fe-402e-841b-abd0014266a2-kube-api-access-qg9j9\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.933418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.932878    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vln9n\" (UniqueName: \"kubernetes.io/projected/62e0637e-89e4-4f94-90e0-12524247f76c-kube-api-access-vln9n\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.933418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.932889    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fc25c-a8fe-402e-841b-abd0014266a2-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:20.933418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:20.932897    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e0637e-89e4-4f94-90e0-12524247f76c-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:21.216190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.216158    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" path="/var/lib/kubelet/pods/6d8cc6c8-8f40-40af-bc28-b3699dab3c88/volumes"
Oct 30 20:22:21.216722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.216706    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" path="/var/lib/kubelet/pods/c1731030-822c-4017-b1e5-62c7a8d013fa/volumes"
Oct 30 20:22:21.217151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.217137    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf801a67-90b1-408d-a14b-86907900d29d" path="/var/lib/kubelet/pods/cf801a67-90b1-408d-a14b-86907900d29d/volumes"
Oct 30 20:22:21.218402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.218373    6058 patch_prober.go:29] interesting pod/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:22:21.218402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.218425    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 20:22:21.449924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.449888    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"
Oct 30 20:22:21.449924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.449917    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd" event={"ID":"62e0637e-89e4-4f94-90e0-12524247f76c","Type":"ContainerDied","Data":"7e13c11638df526182311aef1cdb1994e4b5c30f6de70ad9cd6082a5065cf915"}
Oct 30 20:22:21.450373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.449970    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e13c11638df526182311aef1cdb1994e4b5c30f6de70ad9cd6082a5065cf915"
Oct 30 20:22:21.455527 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.455499    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc" event={"ID":"a42fc25c-a8fe-402e-841b-abd0014266a2","Type":"ContainerDied","Data":"4dbdd7cb80493c97099563a5997f19b359e6814ce973095ab5e6a484d73cb7f1"}
Oct 30 20:22:21.455726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.455705    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dbdd7cb80493c97099563a5997f19b359e6814ce973095ab5e6a484d73cb7f1"
Oct 30 20:22:21.455726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.455505    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"
Oct 30 20:22:21.457223 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.457197    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2" event={"ID":"4b29394f-045e-495b-ab74-adaadec9f485","Type":"ContainerDied","Data":"0feba03ea538d65424e9d7b952cc1f57b836727e9ed0fcdaf0659bff9aeb6391"}
Oct 30 20:22:21.457337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.457241    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0feba03ea538d65424e9d7b952cc1f57b836727e9ed0fcdaf0659bff9aeb6391"
Oct 30 20:22:21.457337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:21.457291    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"
Oct 30 20:22:22.634668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:22.634615    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:22.635196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:22.635118    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:22.636082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:22.636061    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:24.962769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.962738    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"]
Oct 30 20:22:24.963479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963460    6058 cpu_manager.go:414] "containerMap: removing container" podUID="4b29394f-045e-495b-ab74-adaadec9f485" containerName="revoke-keys"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963480    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b29394f-045e-495b-ab74-adaadec9f485" containerName="revoke-keys"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963497    6058 cpu_manager.go:414] "containerMap: removing container" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963502    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963517    6058 cpu_manager.go:414] "containerMap: removing container" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963524    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963541    6058 cpu_manager.go:414] "containerMap: removing container" podUID="62e0637e-89e4-4f94-90e0-12524247f76c" containerName="revoke-keys"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963547    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e0637e-89e4-4f94-90e0-12524247f76c" containerName="revoke-keys"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963553    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a42fc25c-a8fe-402e-841b-abd0014266a2" containerName="revoke-keys"
Oct 30 20:22:24.963558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963558    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42fc25c-a8fe-402e-841b-abd0014266a2" containerName="revoke-keys"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963571    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963575    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963653    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8cc6c8-8f40-40af-bc28-b3699dab3c88" containerName="main"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963663    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1731030-822c-4017-b1e5-62c7a8d013fa" containerName="main"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963673    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b29394f-045e-495b-ab74-adaadec9f485" containerName="revoke-keys"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963680    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e0637e-89e4-4f94-90e0-12524247f76c" containerName="revoke-keys"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963685    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42fc25c-a8fe-402e-841b-abd0014266a2" containerName="revoke-keys"
Oct 30 20:22:24.963821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.963691    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf801a67-90b1-408d-a14b-86907900d29d" containerName="main"
Oct 30 20:22:24.968738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.968716    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:24.976516 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.976493    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:24.976639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.976511    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"]
Oct 30 20:22:24.976778 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.976758    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:24.976879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.976795    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:22:24.976997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.976931    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:24.992552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.991986    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"]
Oct 30 20:22:24.998956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:24.997425    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.015869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.015304    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"]
Oct 30 20:22:25.105055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.105026    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnrk\" (UniqueName: \"kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.105203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.105098    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8m85\" (UniqueName: \"kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.105248 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.105197    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.105283 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.105248    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.105345 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.105325    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.105386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.105377    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.206801 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.206764    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8m85\" (UniqueName: \"kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.207077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.206837    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.207077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.206879    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.207077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.206906    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.207077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.206965    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.207320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.207134    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnrk\" (UniqueName: \"kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.207690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.207663    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.207802 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.207727    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.209722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.209692    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.209997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.209973    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.220171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.220106    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnrk\" (UniqueName: \"kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk\") pod \"maas-api-revoke-sub-ee67d537-pkmrh\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.220363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.220341    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8m85\" (UniqueName: \"kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85\") pod \"maas-api-revoke-sub-acc82938-r5ntp\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.291159 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.291129    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:25.324794 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.324769    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:25.391317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.390582    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" containerID="cri-o://b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4" gracePeriod=45
Oct 30 20:22:25.557494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.557391    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 20:22:25.588708 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:25.588676    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a3c647_5851_4cd4_a366_a154129abc8e.slice/crio-37510eecb65bd5b306a3ed379aee22780a061cc25f90e18065bf2310324e6776 WatchSource:0}: Error finding container 37510eecb65bd5b306a3ed379aee22780a061cc25f90e18065bf2310324e6776: Status 404 returned error can't find the container with id 37510eecb65bd5b306a3ed379aee22780a061cc25f90e18065bf2310324e6776
Oct 30 20:22:25.589404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.589374    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"]
Oct 30 20:22:25.685392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.685358    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"]
Oct 30 20:22:25.701519 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.701497    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:22:25.823200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823122    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs\") pod \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") "
Oct 30 20:22:25.823200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823162    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home\") pod \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") "
Oct 30 20:22:25.823200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823187    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj\" (UniqueName: \"kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj\") pod \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") "
Oct 30 20:22:25.823467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823235    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache\") pod \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") "
Oct 30 20:22:25.823467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823286    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir\") pod \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") "
Oct 30 20:22:25.823467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823404    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm\") pod \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\" (UID: \"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed\") "
Oct 30 20:22:25.823467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823437    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home" pod "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" (UID: "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:25.823661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823549    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir" pod "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" (UID: "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:25.823661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.823567    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache" pod "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" (UID: "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:25.824185 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.824153    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:25.824185 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.824185    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:25.824381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.824202    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:25.825399 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.825364    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs" pod "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" (UID: "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:25.825501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.825481    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj" pod "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" (UID: "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed"). InnerVolumeSpecName "kube-api-access-7v7sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:25.825538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.825508    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm" pod "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" (UID: "c2d5ac13-acaa-4baf-bfba-c38fb728d8ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:25.848759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.848733    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing" containerID="cri-o://6d7500cdaba04156b05daa20ecb354fbd0cfd1c4237e73139ed3286023765231" gracePeriod=25
Oct 30 20:22:25.925509 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.925473    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:25.925509 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.925503    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:25.925509 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:25.925512    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v7sj\" (UniqueName: \"kubernetes.io/projected/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed-kube-api-access-7v7sj\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.108870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.108775    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:26.108870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.108849    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:26.109313 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.109070    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:26.521057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.520949    6058 generic.go:356] "Generic (PLEG): container finished" podID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerID="b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4" exitCode=0
Oct 30 20:22:26.521057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.520994    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"
Oct 30 20:22:26.521057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.521037    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" event={"ID":"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed","Type":"ContainerDied","Data":"b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4"}
Oct 30 20:22:26.521288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.521085    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d" event={"ID":"c2d5ac13-acaa-4baf-bfba-c38fb728d8ed","Type":"ContainerDied","Data":"3b364f744d26485b1985ee4f90b929506a06806f71804d99b7519b2c001261ec"}
Oct 30 20:22:26.521288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.521104    6058 scope.go:122] "RemoveContainer" containerID="b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4"
Oct 30 20:22:26.523148 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.523115    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh" event={"ID":"a49d1f64-f2f4-4d21-938b-16fd7781b8ec","Type":"ContainerStarted","Data":"f587ea1735cfd130ae2e9af6d3436aad4c9344bdba4ebd88dbc7f8306dee7b46"}
Oct 30 20:22:26.523302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.523165    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh" event={"ID":"a49d1f64-f2f4-4d21-938b-16fd7781b8ec","Type":"ContainerStarted","Data":"6fa5b1189c90b49c0184cd179af2ec861f0c24ebb59df3a2278226d7c3b982cf"}
Oct 30 20:22:26.524950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.524924    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp" event={"ID":"71a3c647-5851-4cd4-a366-a154129abc8e","Type":"ContainerStarted","Data":"64123ff8be9619801ed76d3eb0ef3b9d25ffd505e80bddcacbea1638615ba854"}
Oct 30 20:22:26.525082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.524970    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp" event={"ID":"71a3c647-5851-4cd4-a366-a154129abc8e","Type":"ContainerStarted","Data":"37510eecb65bd5b306a3ed379aee22780a061cc25f90e18065bf2310324e6776"}
Oct 30 20:22:26.533230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.533213    6058 scope.go:122] "RemoveContainer" containerID="b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4"
Oct 30 20:22:26.533471 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:26.533450    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4\": container with ID starting with b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4 not found: ID does not exist" containerID="b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4"
Oct 30 20:22:26.533527 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.533481    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4"} err="failed to get container status \"b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4\": rpc error: code = NotFound desc = could not find container \"b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4\": container with ID starting with b0c5e487e06dc715d8b7d5c50b33887d194060ceaae9678211726686e615bff4 not found: ID does not exist"
Oct 30 20:22:26.547128 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.547088    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp" podStartSLOduration=2.5470774670000003 podStartE2EDuration="2.547077467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:26.543681842 +0000 UTC m=+1079.447101871" watchObservedRunningTime="2026-10-30 20:22:26.547077467 +0000 UTC m=+1079.450497492"
Oct 30 20:22:26.566175 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.566130    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh" podStartSLOduration=2.566114588 podStartE2EDuration="2.566114588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:26.561925311 +0000 UTC m=+1079.465345339" watchObservedRunningTime="2026-10-30 20:22:26.566114588 +0000 UTC m=+1079.469534616"
Oct 30 20:22:26.582355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.582333    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"]
Oct 30 20:22:26.588771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:26.588748    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9c901b/simulator-w4-9c901b-kserve-7b797b8f4f-b7q4d"]
Oct 30 20:22:27.214960 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:27.214928    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" path="/var/lib/kubelet/pods/c2d5ac13-acaa-4baf-bfba-c38fb728d8ed/volumes"
Oct 30 20:22:27.535507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:27.535420    6058 generic.go:356] "Generic (PLEG): container finished" podID="a49d1f64-f2f4-4d21-938b-16fd7781b8ec" containerID="f587ea1735cfd130ae2e9af6d3436aad4c9344bdba4ebd88dbc7f8306dee7b46" exitCode=0
Oct 30 20:22:27.535507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:27.535490    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh" event={"ID":"a49d1f64-f2f4-4d21-938b-16fd7781b8ec","Type":"ContainerDied","Data":"f587ea1735cfd130ae2e9af6d3436aad4c9344bdba4ebd88dbc7f8306dee7b46"}
Oct 30 20:22:27.537155 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:27.537130    6058 generic.go:356] "Generic (PLEG): container finished" podID="71a3c647-5851-4cd4-a366-a154129abc8e" containerID="64123ff8be9619801ed76d3eb0ef3b9d25ffd505e80bddcacbea1638615ba854" exitCode=0
Oct 30 20:22:27.537265 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:27.537194    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp" event={"ID":"71a3c647-5851-4cd4-a366-a154129abc8e","Type":"ContainerDied","Data":"64123ff8be9619801ed76d3eb0ef3b9d25ffd505e80bddcacbea1638615ba854"}
Oct 30 20:22:28.784212 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.783836    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:28.785830 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.785806    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:28.887902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.887861    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85\" (UniqueName: \"kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85\") pod \"71a3c647-5851-4cd4-a366-a154129abc8e\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") "
Oct 30 20:22:28.888123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.887933    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca\") pod \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") "
Oct 30 20:22:28.888123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.888001    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca\") pod \"71a3c647-5851-4cd4-a366-a154129abc8e\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") "
Oct 30 20:22:28.888123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.888112    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca\") pod \"71a3c647-5851-4cd4-a366-a154129abc8e\" (UID: \"71a3c647-5851-4cd4-a366-a154129abc8e\") "
Oct 30 20:22:28.888298 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.888195    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk\" (UniqueName: \"kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk\") pod \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") "
Oct 30 20:22:28.888298 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.888276    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca\") pod \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\" (UID: \"a49d1f64-f2f4-4d21-938b-16fd7781b8ec\") "
Oct 30 20:22:28.888606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.888534    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca" pod "71a3c647-5851-4cd4-a366-a154129abc8e" (UID: "71a3c647-5851-4cd4-a366-a154129abc8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:28.888785 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.888758    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca" pod "a49d1f64-f2f4-4d21-938b-16fd7781b8ec" (UID: "a49d1f64-f2f4-4d21-938b-16fd7781b8ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:28.889090 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.889038    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:28.889090 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.889067    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:28.890440 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.890416    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca" pod "71a3c647-5851-4cd4-a366-a154129abc8e" (UID: "71a3c647-5851-4cd4-a366-a154129abc8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:28.890667 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.890645    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85" pod "71a3c647-5851-4cd4-a366-a154129abc8e" (UID: "71a3c647-5851-4cd4-a366-a154129abc8e"). InnerVolumeSpecName "kube-api-access-l8m85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:28.890736 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.890704    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk" pod "a49d1f64-f2f4-4d21-938b-16fd7781b8ec" (UID: "a49d1f64-f2f4-4d21-938b-16fd7781b8ec"). InnerVolumeSpecName "kube-api-access-bvnrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:28.891111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.891092    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca" pod "a49d1f64-f2f4-4d21-938b-16fd7781b8ec" (UID: "a49d1f64-f2f4-4d21-938b-16fd7781b8ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:28.990870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.990828    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnrk\" (UniqueName: \"kubernetes.io/projected/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-kube-api-access-bvnrk\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:28.990870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.990868    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8m85\" (UniqueName: \"kubernetes.io/projected/71a3c647-5851-4cd4-a366-a154129abc8e-kube-api-access-l8m85\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:28.991135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.990885    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d1f64-f2f4-4d21-938b-16fd7781b8ec-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:28.991135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:28.990900    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a3c647-5851-4cd4-a366-a154129abc8e-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:29.554919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:29.554838    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"
Oct 30 20:22:29.554919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:29.554855    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh" event={"ID":"a49d1f64-f2f4-4d21-938b-16fd7781b8ec","Type":"ContainerDied","Data":"6fa5b1189c90b49c0184cd179af2ec861f0c24ebb59df3a2278226d7c3b982cf"}
Oct 30 20:22:29.554919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:29.554912    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa5b1189c90b49c0184cd179af2ec861f0c24ebb59df3a2278226d7c3b982cf"
Oct 30 20:22:29.556497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:29.556475    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"
Oct 30 20:22:29.556640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:29.556466    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp" event={"ID":"71a3c647-5851-4cd4-a366-a154129abc8e","Type":"ContainerDied","Data":"37510eecb65bd5b306a3ed379aee22780a061cc25f90e18065bf2310324e6776"}
Oct 30 20:22:29.556640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:29.556604    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37510eecb65bd5b306a3ed379aee22780a061cc25f90e18065bf2310324e6776"
Oct 30 20:22:30.027304 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.027261    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main" containerID="cri-o://d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5" gracePeriod=44
Oct 30 20:22:30.322919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.322894    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:22:30.325130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325105    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg\" (UniqueName: \"kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg\") pod \"4eaf0ace-20fd-4aca-a952-13e700efe595\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") "
Oct 30 20:22:30.325278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325158    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home\") pod \"4eaf0ace-20fd-4aca-a952-13e700efe595\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") "
Oct 30 20:22:30.325278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325233    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs\") pod \"4eaf0ace-20fd-4aca-a952-13e700efe595\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") "
Oct 30 20:22:30.325278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325262    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir\") pod \"4eaf0ace-20fd-4aca-a952-13e700efe595\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") "
Oct 30 20:22:30.325464 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325336    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache\") pod \"4eaf0ace-20fd-4aca-a952-13e700efe595\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") "
Oct 30 20:22:30.325579 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325549    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir" pod "4eaf0ace-20fd-4aca-a952-13e700efe595" (UID: "4eaf0ace-20fd-4aca-a952-13e700efe595"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:30.325632 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325584    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache" pod "4eaf0ace-20fd-4aca-a952-13e700efe595" (UID: "4eaf0ace-20fd-4aca-a952-13e700efe595"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:30.325685 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325672    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home" pod "4eaf0ace-20fd-4aca-a952-13e700efe595" (UID: "4eaf0ace-20fd-4aca-a952-13e700efe595"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:30.325739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.325677    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm\") pod \"4eaf0ace-20fd-4aca-a952-13e700efe595\" (UID: \"4eaf0ace-20fd-4aca-a952-13e700efe595\") "
Oct 30 20:22:30.326307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.326285    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:30.326307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.326306    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:30.326484 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.326315    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:30.327491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.327468    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg" pod "4eaf0ace-20fd-4aca-a952-13e700efe595" (UID: "4eaf0ace-20fd-4aca-a952-13e700efe595"). InnerVolumeSpecName "kube-api-access-l55dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:30.327770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.327734    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs" pod "4eaf0ace-20fd-4aca-a952-13e700efe595" (UID: "4eaf0ace-20fd-4aca-a952-13e700efe595"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:30.327908 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.327888    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm" pod "4eaf0ace-20fd-4aca-a952-13e700efe595" (UID: "4eaf0ace-20fd-4aca-a952-13e700efe595"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:30.427911 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.427875    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55dg\" (UniqueName: \"kubernetes.io/projected/4eaf0ace-20fd-4aca-a952-13e700efe595-kube-api-access-l55dg\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:30.427911 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.427902    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eaf0ace-20fd-4aca-a952-13e700efe595-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:30.427911 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.427912    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eaf0ace-20fd-4aca-a952-13e700efe595-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:30.463451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.463409    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"]
Oct 30 20:22:30.543160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.543127    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"]
Oct 30 20:22:30.544744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544709    6058 cpu_manager.go:414] "containerMap: removing container" podUID="71a3c647-5851-4cd4-a366-a154129abc8e" containerName="revoke-keys"
Oct 30 20:22:30.544744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544743    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a3c647-5851-4cd4-a366-a154129abc8e" containerName="revoke-keys"
Oct 30 20:22:30.544925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544792    6058 cpu_manager.go:414] "containerMap: removing container" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main"
Oct 30 20:22:30.544925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544801    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main"
Oct 30 20:22:30.544925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544863    6058 cpu_manager.go:414] "containerMap: removing container" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main"
Oct 30 20:22:30.544925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544876    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main"
Oct 30 20:22:30.544925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544888    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a49d1f64-f2f4-4d21-938b-16fd7781b8ec" containerName="revoke-keys"
Oct 30 20:22:30.544925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.544895    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49d1f64-f2f4-4d21-938b-16fd7781b8ec" containerName="revoke-keys"
Oct 30 20:22:30.545198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.545174    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerName="main"
Oct 30 20:22:30.545244 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.545197    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49d1f64-f2f4-4d21-938b-16fd7781b8ec" containerName="revoke-keys"
Oct 30 20:22:30.545244 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.545210    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a3c647-5851-4cd4-a366-a154129abc8e" containerName="revoke-keys"
Oct 30 20:22:30.545244 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.545229    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d5ac13-acaa-4baf-bfba-c38fb728d8ed" containerName="main"
Oct 30 20:22:30.550765 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.550694    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.556648 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.556623    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"]
Oct 30 20:22:30.557423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.557400    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:22:30.559480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.557962    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:30.559480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.558052    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:30.559480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.558314    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:30.569739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.569067    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.584491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.584457    6058 generic.go:356] "Generic (PLEG): container finished" podID="4eaf0ace-20fd-4aca-a952-13e700efe595" containerID="d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5" exitCode=0
Oct 30 20:22:30.584769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.584740    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" event={"ID":"4eaf0ace-20fd-4aca-a952-13e700efe595","Type":"ContainerDied","Data":"d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5"}
Oct 30 20:22:30.584901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.584786    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z" event={"ID":"4eaf0ace-20fd-4aca-a952-13e700efe595","Type":"ContainerDied","Data":"c16e681a2de1556b0cbe81fe65f67fe5d1cc379611c4b20fb14abf8e5109a2c1"}
Oct 30 20:22:30.584901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.584810    6058 scope.go:122] "RemoveContainer" containerID="d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5"
Oct 30 20:22:30.585071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.585056    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"
Oct 30 20:22:30.607514 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.606554    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"]
Oct 30 20:22:30.612370 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.612345    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"]
Oct 30 20:22:30.616922 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.616900    6058 scope.go:122] "RemoveContainer" containerID="d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5"
Oct 30 20:22:30.617283 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:30.617256    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5\": container with ID starting with d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5 not found: ID does not exist" containerID="d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5"
Oct 30 20:22:30.617354 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.617296    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5"} err="failed to get container status \"d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5\": rpc error: code = NotFound desc = could not find container \"d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5\": container with ID starting with d12ef3ce9c0062be63376eb2bc3abeb0c10f7dc07d8e307d0cb89fe305b0bda5 not found: ID does not exist"
Oct 30 20:22:30.629337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.629305    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"]
Oct 30 20:22:30.630147 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.630125    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.630250 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.630176    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.630250 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.630219    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68fg\" (UniqueName: \"kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.630250 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.630241    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzfb\" (UniqueName: \"kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.630431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.630317    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.630431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.630366    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.638117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.638096    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-44135b/premium-w5-44135b-kserve-64b677c996-x456z"]
Oct 30 20:22:30.731872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.731835    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.732041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.731909    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.732113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.732061    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.732167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.732120    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.732219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.732169    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68fg\" (UniqueName: \"kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.732219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.732197    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzfb\" (UniqueName: \"kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.732803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.732776    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.732882 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.732842    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.734363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.734342    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.734448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.734365    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.745816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.745790    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzfb\" (UniqueName: \"kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb\") pod \"maas-api-revoke-sub-1f62ea98-27szt\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.745980 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.745965    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68fg\" (UniqueName: \"kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg\") pod \"maas-api-revoke-sub-54cadc29-kr44l\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:30.898695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.898665    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:30.915316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:30.915291    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:31.108114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.107704    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:22:31.175187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.175091    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:22:31.222103 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.219407    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eaf0ace-20fd-4aca-a952-13e700efe595" path="/var/lib/kubelet/pods/4eaf0ace-20fd-4aca-a952-13e700efe595/volumes"
Oct 30 20:22:31.227342 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.226603    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"]
Oct 30 20:22:31.242351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.239644    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp\" (UniqueName: \"kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp\") pod \"edd00c5f-7a1e-45b2-a6d0-10471794c402\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") "
Oct 30 20:22:31.242351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.239789    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume\") pod \"edd00c5f-7a1e-45b2-a6d0-10471794c402\" (UID: \"edd00c5f-7a1e-45b2-a6d0-10471794c402\") "
Oct 30 20:22:31.242351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.241849    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume" pod "edd00c5f-7a1e-45b2-a6d0-10471794c402" (UID: "edd00c5f-7a1e-45b2-a6d0-10471794c402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:31.256297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.253537    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"]
Oct 30 20:22:31.273897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.260845    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp" pod "edd00c5f-7a1e-45b2-a6d0-10471794c402" (UID: "edd00c5f-7a1e-45b2-a6d0-10471794c402"). InnerVolumeSpecName "kube-api-access-bdhzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:31.342696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.342641    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdhzp\" (UniqueName: \"kubernetes.io/projected/edd00c5f-7a1e-45b2-a6d0-10471794c402-kube-api-access-bdhzp\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:31.343187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.343156    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd00c5f-7a1e-45b2-a6d0-10471794c402-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:31.597417 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.597385    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt" event={"ID":"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690","Type":"ContainerStarted","Data":"e793e9b23c5b7e052e1ea2814548839c08ed1fa933e85cbe32339209469941a7"}
Oct 30 20:22:31.597582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.597440    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt" event={"ID":"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690","Type":"ContainerStarted","Data":"cd54895383a396f2cc90dea1ae3067357bf1c02c4c846470ec553f8d58503a98"}
Oct 30 20:22:31.599262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.599232    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l" event={"ID":"9b29b091-7e73-4680-8df9-64dfca7f537d","Type":"ContainerStarted","Data":"89fa316a9b021cf518fd0cdc75e2a7b87d0abe67787d9c5c13047b4428106dad"}
Oct 30 20:22:31.600769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.600739    6058 generic.go:356] "Generic (PLEG): container finished" podID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerID="1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6" exitCode=0
Oct 30 20:22:31.600909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.600786    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"
Oct 30 20:22:31.600909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.600836    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" event={"ID":"edd00c5f-7a1e-45b2-a6d0-10471794c402","Type":"ContainerDied","Data":"1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6"}
Oct 30 20:22:31.600909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.600884    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54" event={"ID":"edd00c5f-7a1e-45b2-a6d0-10471794c402","Type":"ContainerDied","Data":"80f5e894724df5baf885cab2c53da618f3415298d4644f81a7573faedd1d2be9"}
Oct 30 20:22:31.601082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.600915    6058 scope.go:122] "RemoveContainer" containerID="1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6"
Oct 30 20:22:31.612187 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.612168    6058 scope.go:122] "RemoveContainer" containerID="1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6"
Oct 30 20:22:31.612475 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:31.612454    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6\": container with ID starting with 1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6 not found: ID does not exist" containerID="1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6"
Oct 30 20:22:31.612546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.612487    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6"} err="failed to get container status \"1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6\": rpc error: code = NotFound desc = could not find container \"1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6\": container with ID starting with 1c1d0eb028f9e48c6d536ebfd4b8034a8322c7ce57afe51123f08886be8004e6 not found: ID does not exist"
Oct 30 20:22:31.622133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.622082    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt" podStartSLOduration=1.622066206 podStartE2EDuration="1.622066206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:31.618117566 +0000 UTC m=+1084.521537592" watchObservedRunningTime="2026-10-30 20:22:31.622066206 +0000 UTC m=+1084.525486233"
Oct 30 20:22:31.636436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.636409    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"]
Oct 30 20:22:31.643466 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.643441    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c646da-8464897474-nxf54"]
Oct 30 20:22:31.938067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.938034    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"]
Oct 30 20:22:31.938913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.938884    6058 cpu_manager.go:414] "containerMap: removing container" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerName="payload-processing"
Oct 30 20:22:31.938913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.938910    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerName="payload-processing"
Oct 30 20:22:31.939141 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.939094    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" containerName="payload-processing"
Oct 30 20:22:31.947054 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.946400    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:31.979465 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:31.979429    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"]
Oct 30 20:22:32.061637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.061551    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.061637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.061611    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.061866 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.061763    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6th\" (UniqueName: \"kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.163243 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.163207    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.163633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.163274    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.163633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.163434    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6th\" (UniqueName: \"kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.164121 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.164094    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.165721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.165700    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.181240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.181215    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6th\" (UniqueName: \"kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th\") pod \"maas-api-revoke-sub-7d19f1e0-pvwdc\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.264110 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.264014    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"]
Oct 30 20:22:32.270191 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.270165    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c646da-75gx9"]
Oct 30 20:22:32.276228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.276206    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:32.989482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:32.989403    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"]
Oct 30 20:22:33.013723 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.013689    6058 generic.go:356] "Generic (PLEG): container finished" podID="9b29b091-7e73-4680-8df9-64dfca7f537d" containerID="97b0a1322ea2d14c69ec418865223364260377bd3f4e4943626ffa236075fca9" exitCode=0
Oct 30 20:22:33.013978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.013953    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l" event={"ID":"9b29b091-7e73-4680-8df9-64dfca7f537d","Type":"ContainerDied","Data":"97b0a1322ea2d14c69ec418865223364260377bd3f4e4943626ffa236075fca9"}
Oct 30 20:22:33.156038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.150951    6058 generic.go:356] "Generic (PLEG): container finished" podID="4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" containerID="e793e9b23c5b7e052e1ea2814548839c08ed1fa933e85cbe32339209469941a7" exitCode=0
Oct 30 20:22:33.165031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.154454    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt" event={"ID":"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690","Type":"ContainerDied","Data":"e793e9b23c5b7e052e1ea2814548839c08ed1fa933e85cbe32339209469941a7"}
Oct 30 20:22:33.373038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.369779    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf" path="/var/lib/kubelet/pods/60380d37-c02c-4aa5-bf6c-b45f4fe3c7cf/volumes"
Oct 30 20:22:33.373038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.370497    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd00c5f-7a1e-45b2-a6d0-10471794c402" path="/var/lib/kubelet/pods/edd00c5f-7a1e-45b2-a6d0-10471794c402/volumes"
Oct 30 20:22:33.830216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:33.803430    6058 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:34.164036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.137445    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:34.336829 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.145092    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:34.337716 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.308730    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc" event={"ID":"14c261b9-b134-4528-b9de-3e64428aa26c","Type":"ContainerStarted","Data":"ca73030547edf79878c76d9edfa10b48a3b3723565db2f1ffb74b9198a0e3fdd"}
Oct 30 20:22:34.337946 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.337915    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc" event={"ID":"14c261b9-b134-4528-b9de-3e64428aa26c","Type":"ContainerStarted","Data":"057b09a1b7eee8d3dcdd74cd3513492db611699d554dc6e3ab86707cc706e544"}
Oct 30 20:22:34.375045 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.372638    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc" podStartSLOduration=3.372616584 podStartE2EDuration="3.372616584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:34.356103442 +0000 UTC m=+1087.259523464" watchObservedRunningTime="2026-10-30 20:22:34.372616584 +0000 UTC m=+1087.276036610"
Oct 30 20:22:34.411103 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.411057    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:34.930031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.917159    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:22:34.930031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.917233    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:34.978027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.974068    6058 patch_prober.go:29] interesting pod/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:22:34.978027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:34.974164    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:35.419117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.376369    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:35.419117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.392840    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n" podUID="acb420cc-2386-4cca-b3ae-a53a9fb370e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 30 20:22:35.419117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.418092    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:35.603032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.578105    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:22:35.783379 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.783348    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:22:35.783578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.783565    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:22:35.931960 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.930672    6058 generic.go:356] "Generic (PLEG): container finished" podID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerID="e4bc98ed92643d0df4bab799b255d7453b6ca550191d87ca2962bbb58f7318be" exitCode=0
Oct 30 20:22:35.932259 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.932235    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" event={"ID":"7fcf388f-663e-47a2-9cd7-d38c9b158a4c","Type":"ContainerDied","Data":"e4bc98ed92643d0df4bab799b255d7453b6ca550191d87ca2962bbb58f7318be"}
Oct 30 20:22:35.935546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.935521    6058 generic.go:356] "Generic (PLEG): container finished" podID="14c261b9-b134-4528-b9de-3e64428aa26c" containerID="ca73030547edf79878c76d9edfa10b48a3b3723565db2f1ffb74b9198a0e3fdd" exitCode=0
Oct 30 20:22:35.935859 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:35.935840    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc" event={"ID":"14c261b9-b134-4528-b9de-3e64428aa26c","Type":"ContainerDied","Data":"ca73030547edf79878c76d9edfa10b48a3b3723565db2f1ffb74b9198a0e3fdd"}
Oct 30 20:22:36.304118 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.298495    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:36.321048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.315563    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:36.349575 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.349543    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n"
Oct 30 20:22:36.352374 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.352353    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:36.409571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.409500    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:36.421651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.421519    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:22:36.442273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.442245    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:22:36.470172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.453433    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537079    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb\" (UniqueName: \"kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb\") pod \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537156    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume\") pod \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537213    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca\") pod \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537259    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz\" (UniqueName: \"kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz\") pod \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\" (UID: \"7fcf388f-663e-47a2-9cd7-d38c9b158a4c\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537340    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg\" (UniqueName: \"kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg\") pod \"9b29b091-7e73-4680-8df9-64dfca7f537d\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537417    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca\") pod \"9b29b091-7e73-4680-8df9-64dfca7f537d\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537495    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca\") pod \"9b29b091-7e73-4680-8df9-64dfca7f537d\" (UID: \"9b29b091-7e73-4680-8df9-64dfca7f537d\") "
Oct 30 20:22:36.538053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.537528    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca\") pod \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\" (UID: \"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690\") "
Oct 30 20:22:36.540525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.540495    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume" pod "7fcf388f-663e-47a2-9cd7-d38c9b158a4c" (UID: "7fcf388f-663e-47a2-9cd7-d38c9b158a4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:36.546044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.545998    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca" pod "9b29b091-7e73-4680-8df9-64dfca7f537d" (UID: "9b29b091-7e73-4680-8df9-64dfca7f537d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:36.546696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.546671    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca" pod "4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" (UID: "4eb516c6-e0cd-4b82-9a7c-8a13b9d74690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:36.549647 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.549624    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca" pod "9b29b091-7e73-4680-8df9-64dfca7f537d" (UID: "9b29b091-7e73-4680-8df9-64dfca7f537d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:36.549981 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.549953    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb" pod "4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" (UID: "4eb516c6-e0cd-4b82-9a7c-8a13b9d74690"). InnerVolumeSpecName "kube-api-access-hvzfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:36.550476 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.550428    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca" pod "4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" (UID: "4eb516c6-e0cd-4b82-9a7c-8a13b9d74690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:36.551273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.551231    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz" pod "7fcf388f-663e-47a2-9cd7-d38c9b158a4c" (UID: "7fcf388f-663e-47a2-9cd7-d38c9b158a4c"). InnerVolumeSpecName "kube-api-access-22jkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:36.553347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.553324    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg" pod "9b29b091-7e73-4680-8df9-64dfca7f537d" (UID: "9b29b091-7e73-4680-8df9-64dfca7f537d"). InnerVolumeSpecName "kube-api-access-m68fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.755882    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m68fg\" (UniqueName: \"kubernetes.io/projected/9b29b091-7e73-4680-8df9-64dfca7f537d-kube-api-access-m68fg\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.755917    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.755935    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b29b091-7e73-4680-8df9-64dfca7f537d-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.755951    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.755971    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzfb\" (UniqueName: \"kubernetes.io/projected/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-kube-api-access-hvzfb\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.755986    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.756019    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.756172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.756038    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jkz\" (UniqueName: \"kubernetes.io/projected/7fcf388f-663e-47a2-9cd7-d38c9b158a4c-kube-api-access-22jkz\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:36.863607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.863575    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:22:36.953121 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.953087    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" event={"ID":"7fcf388f-663e-47a2-9cd7-d38c9b158a4c","Type":"ContainerDied","Data":"6ccfc84fc04aa931cb13b2d2c4f198269bd9663e6b48086c299a8af173151b85"}
Oct 30 20:22:36.953301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.953157    6058 scope.go:122] "RemoveContainer" containerID="e4bc98ed92643d0df4bab799b255d7453b6ca550191d87ca2962bbb58f7318be"
Oct 30 20:22:36.953301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.953159    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"
Oct 30 20:22:36.955614 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.955599    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"
Oct 30 20:22:36.956481 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.955702    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt" event={"ID":"4eb516c6-e0cd-4b82-9a7c-8a13b9d74690","Type":"ContainerDied","Data":"cd54895383a396f2cc90dea1ae3067357bf1c02c4c846470ec553f8d58503a98"}
Oct 30 20:22:36.956566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.956511    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd54895383a396f2cc90dea1ae3067357bf1c02c4c846470ec553f8d58503a98"
Oct 30 20:22:36.958692 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.958649    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l" event={"ID":"9b29b091-7e73-4680-8df9-64dfca7f537d","Type":"ContainerDied","Data":"89fa316a9b021cf518fd0cdc75e2a7b87d0abe67787d9c5c13047b4428106dad"}
Oct 30 20:22:36.958796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.958698    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89fa316a9b021cf518fd0cdc75e2a7b87d0abe67787d9c5c13047b4428106dad"
Oct 30 20:22:36.958796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:36.958705    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"
Oct 30 20:22:37.019137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.019060    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"]
Oct 30 20:22:37.030506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.028283    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb"]
Oct 30 20:22:37.114767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.114719    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-75e734-b6f8799c6-qv5bb" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.40:50052\" within 1s: context deadline exceeded"
Oct 30 20:22:37.114949 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:37.114795    6058 logging.go:55] [core] [Channel #546 SubChannel #549]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.40:50052", ServerName: "10.133.0.40:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.40:50052: operation was canceled"
Oct 30 20:22:37.156983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.156958    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:37.167536 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.167512    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th\" (UniqueName: \"kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th\") pod \"14c261b9-b134-4528-b9de-3e64428aa26c\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") "
Oct 30 20:22:37.167678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.167565    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca\") pod \"14c261b9-b134-4528-b9de-3e64428aa26c\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") "
Oct 30 20:22:37.167678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.167600    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca\") pod \"14c261b9-b134-4528-b9de-3e64428aa26c\" (UID: \"14c261b9-b134-4528-b9de-3e64428aa26c\") "
Oct 30 20:22:37.168038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.167998    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca" pod "14c261b9-b134-4528-b9de-3e64428aa26c" (UID: "14c261b9-b134-4528-b9de-3e64428aa26c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:37.168342 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.168314    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:37.169831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.169808    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca" pod "14c261b9-b134-4528-b9de-3e64428aa26c" (UID: "14c261b9-b134-4528-b9de-3e64428aa26c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:37.169916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.169808    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th" pod "14c261b9-b134-4528-b9de-3e64428aa26c" (UID: "14c261b9-b134-4528-b9de-3e64428aa26c"). InnerVolumeSpecName "kube-api-access-pq6th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:37.214319 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.214292    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" path="/var/lib/kubelet/pods/7fcf388f-663e-47a2-9cd7-d38c9b158a4c/volumes"
Oct 30 20:22:37.269303 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.269231    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq6th\" (UniqueName: \"kubernetes.io/projected/14c261b9-b134-4528-b9de-3e64428aa26c-kube-api-access-pq6th\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:37.269303 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.269261    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c261b9-b134-4528-b9de-3e64428aa26c-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:37.423263 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.423181    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"]
Oct 30 20:22:37.972770 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.972733    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc" event={"ID":"14c261b9-b134-4528-b9de-3e64428aa26c","Type":"ContainerDied","Data":"057b09a1b7eee8d3dcdd74cd3513492db611699d554dc6e3ab86707cc706e544"}
Oct 30 20:22:37.972976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.972795    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="057b09a1b7eee8d3dcdd74cd3513492db611699d554dc6e3ab86707cc706e544"
Oct 30 20:22:37.972976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:37.972863    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"
Oct 30 20:22:41.274041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.271761    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"]
Oct 30 20:22:41.274041 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.271981    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy" containerID="cri-o://864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19" gracePeriod=30
Oct 30 20:22:41.285061 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.282137    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"]
Oct 30 20:22:41.285061 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.282359    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy" containerID="cri-o://62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664" gracePeriod=30
Oct 30 20:22:41.289221 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.289191    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"]
Oct 30 20:22:41.289439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.289415    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy" containerID="cri-o://7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037" gracePeriod=30
Oct 30 20:22:41.984139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984104    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"]
Oct 30 20:22:41.984814 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984794    6058 cpu_manager.go:414] "containerMap: removing container" podUID="4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" containerName="revoke-keys"
Oct 30 20:22:41.984858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984818    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" containerName="revoke-keys"
Oct 30 20:22:41.984858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984836    6058 cpu_manager.go:414] "containerMap: removing container" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing"
Oct 30 20:22:41.984858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984844    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing"
Oct 30 20:22:41.984959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984890    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9b29b091-7e73-4680-8df9-64dfca7f537d" containerName="revoke-keys"
Oct 30 20:22:41.984959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984899    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b29b091-7e73-4680-8df9-64dfca7f537d" containerName="revoke-keys"
Oct 30 20:22:41.984959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984929    6058 cpu_manager.go:414] "containerMap: removing container" podUID="14c261b9-b134-4528-b9de-3e64428aa26c" containerName="revoke-keys"
Oct 30 20:22:41.984959 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.984936    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c261b9-b134-4528-b9de-3e64428aa26c" containerName="revoke-keys"
Oct 30 20:22:41.985114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.985085    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" containerName="revoke-keys"
Oct 30 20:22:41.985114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.985098    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcf388f-663e-47a2-9cd7-d38c9b158a4c" containerName="payload-processing"
Oct 30 20:22:41.985114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.985106    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b29b091-7e73-4680-8df9-64dfca7f537d" containerName="revoke-keys"
Oct 30 20:22:41.985208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.985117    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c261b9-b134-4528-b9de-3e64428aa26c" containerName="revoke-keys"
Oct 30 20:22:41.990036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.989997    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:41.998042 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.997983    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:41.998594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.998572    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:22:41.999335 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.999303    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:41.999508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.999483    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:42.000387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:41.999733    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"]
Oct 30 20:22:42.038156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.038123    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.038318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.038185    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvckf\" (UniqueName: \"kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.038362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.038320    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.139969 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.139928    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.140186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.140164    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.140249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.140237    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvckf\" (UniqueName: \"kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.141043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.140976    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.144341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.144314    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.155666 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.155636    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvckf\" (UniqueName: \"kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf\") pod \"maas-api-revoke-sub-d9fd8f84-mqpdl\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.313430 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.313349    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:42.426140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.424092    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerName="payload-processing" containerID="cri-o://36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61" gracePeriod=25
Oct 30 20:22:42.432492 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.429481    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"]
Oct 30 20:22:42.437143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.437116    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-75e734-p6hjh"]
Oct 30 20:22:42.629039 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:42.628982    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676c126a_f54a_491d_a9e7_def447e917b0.slice/crio-c275160a56c3a1d9594a3adeb11d9bf247a155217a4f942f314dc450e0feb336 WatchSource:0}: Error finding container c275160a56c3a1d9594a3adeb11d9bf247a155217a4f942f314dc450e0feb336: Status 404 returned error can't find the container with id c275160a56c3a1d9594a3adeb11d9bf247a155217a4f942f314dc450e0feb336
Oct 30 20:22:42.629883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:42.629851    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"]
Oct 30 20:22:43.025295 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:43.025262    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl" event={"ID":"676c126a-f54a-491d-a9e7-def447e917b0","Type":"ContainerStarted","Data":"81a4603acd1e4860d6b6c73ad19666561c33105007099a9ec10fea066f200e43"}
Oct 30 20:22:43.025420 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:43.025320    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl" event={"ID":"676c126a-f54a-491d-a9e7-def447e917b0","Type":"ContainerStarted","Data":"c275160a56c3a1d9594a3adeb11d9bf247a155217a4f942f314dc450e0feb336"}
Oct 30 20:22:43.047584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:43.047541    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl" podStartSLOduration=2.047526729 podStartE2EDuration="2.047526729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:43.045629322 +0000 UTC m=+1095.949049351" watchObservedRunningTime="2026-10-30 20:22:43.047526729 +0000 UTC m=+1095.950946756"
Oct 30 20:22:43.217896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:43.217808    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da36db1d-7117-41a6-b347-c66f7eec4921" path="/var/lib/kubelet/pods/da36db1d-7117-41a6-b347-c66f7eec4921/volumes"
Oct 30 20:22:44.034099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:44.034066    6058 generic.go:356] "Generic (PLEG): container finished" podID="676c126a-f54a-491d-a9e7-def447e917b0" containerID="81a4603acd1e4860d6b6c73ad19666561c33105007099a9ec10fea066f200e43" exitCode=0
Oct 30 20:22:44.034562 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:44.034154    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl" event={"ID":"676c126a-f54a-491d-a9e7-def447e917b0","Type":"ContainerDied","Data":"81a4603acd1e4860d6b6c73ad19666561c33105007099a9ec10fea066f200e43"}
Oct 30 20:22:45.351757 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.351717    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:45.375075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.375037    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"]
Oct 30 20:22:45.390978 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.390944    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9c901b-mcqhd"]
Oct 30 20:22:45.405448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.405414    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca\") pod \"676c126a-f54a-491d-a9e7-def447e917b0\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") "
Oct 30 20:22:45.405779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.405748    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf\" (UniqueName: \"kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf\") pod \"676c126a-f54a-491d-a9e7-def447e917b0\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") "
Oct 30 20:22:45.407184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.407163    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca\") pod \"676c126a-f54a-491d-a9e7-def447e917b0\" (UID: \"676c126a-f54a-491d-a9e7-def447e917b0\") "
Oct 30 20:22:45.408585 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.407561    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca" pod "676c126a-f54a-491d-a9e7-def447e917b0" (UID: "676c126a-f54a-491d-a9e7-def447e917b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:45.410404 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.410382    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf" pod "676c126a-f54a-491d-a9e7-def447e917b0" (UID: "676c126a-f54a-491d-a9e7-def447e917b0"). InnerVolumeSpecName "kube-api-access-bvckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:45.417956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.417933    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca" pod "676c126a-f54a-491d-a9e7-def447e917b0" (UID: "676c126a-f54a-491d-a9e7-def447e917b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:45.494055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.491549    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main" containerID="cri-o://bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a" gracePeriod=45
Oct 30 20:22:45.514375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.511565    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:45.514375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.511601    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c126a-f54a-491d-a9e7-def447e917b0-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:45.514375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.511616    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvckf\" (UniqueName: \"kubernetes.io/projected/676c126a-f54a-491d-a9e7-def447e917b0-kube-api-access-bvckf\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:45.826489 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.826451    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:22:45.916699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.916616    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache\") pod \"9cabd8f3-70d4-404e-9adb-7589b76bb013\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") "
Oct 30 20:22:45.916933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.916730    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home\") pod \"9cabd8f3-70d4-404e-9adb-7589b76bb013\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") "
Oct 30 20:22:45.916933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.916820    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w\" (UniqueName: \"kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w\") pod \"9cabd8f3-70d4-404e-9adb-7589b76bb013\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") "
Oct 30 20:22:45.916933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.916895    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir\") pod \"9cabd8f3-70d4-404e-9adb-7589b76bb013\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") "
Oct 30 20:22:45.917156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.916976    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm\") pod \"9cabd8f3-70d4-404e-9adb-7589b76bb013\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") "
Oct 30 20:22:45.917212 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.917187    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs\") pod \"9cabd8f3-70d4-404e-9adb-7589b76bb013\" (UID: \"9cabd8f3-70d4-404e-9adb-7589b76bb013\") "
Oct 30 20:22:45.919525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.919501    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir" pod "9cabd8f3-70d4-404e-9adb-7589b76bb013" (UID: "9cabd8f3-70d4-404e-9adb-7589b76bb013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:45.919864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.919843    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache" pod "9cabd8f3-70d4-404e-9adb-7589b76bb013" (UID: "9cabd8f3-70d4-404e-9adb-7589b76bb013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:45.920000 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.919933    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home" pod "9cabd8f3-70d4-404e-9adb-7589b76bb013" (UID: "9cabd8f3-70d4-404e-9adb-7589b76bb013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:45.922294 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.922261    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs" pod "9cabd8f3-70d4-404e-9adb-7589b76bb013" (UID: "9cabd8f3-70d4-404e-9adb-7589b76bb013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:45.923772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.923743    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm" pod "9cabd8f3-70d4-404e-9adb-7589b76bb013" (UID: "9cabd8f3-70d4-404e-9adb-7589b76bb013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:45.925199 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:45.925168    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w" pod "9cabd8f3-70d4-404e-9adb-7589b76bb013" (UID: "9cabd8f3-70d4-404e-9adb-7589b76bb013"). InnerVolumeSpecName "kube-api-access-j544w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.019391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.019358    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.019391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.019386    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.019391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.019396    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cabd8f3-70d4-404e-9adb-7589b76bb013-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.019596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.019406    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.019596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.019415    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cabd8f3-70d4-404e-9adb-7589b76bb013-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.019596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.019424    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j544w\" (UniqueName: \"kubernetes.io/projected/9cabd8f3-70d4-404e-9adb-7589b76bb013-kube-api-access-j544w\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.059570 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.059327    6058 generic.go:356] "Generic (PLEG): container finished" podID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerID="bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a" exitCode=0
Oct 30 20:22:46.059715 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.059593    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"
Oct 30 20:22:46.069578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.069541    6058 generic.go:356] "Generic (PLEG): container finished" podID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerID="6d7500cdaba04156b05daa20ecb354fbd0cfd1c4237e73139ed3286023765231" exitCode=0
Oct 30 20:22:46.070528 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.070499    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" event={"ID":"9cabd8f3-70d4-404e-9adb-7589b76bb013","Type":"ContainerDied","Data":"bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a"}
Oct 30 20:22:46.070658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.070555    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb" event={"ID":"9cabd8f3-70d4-404e-9adb-7589b76bb013","Type":"ContainerDied","Data":"98a2589cfb44e579b30bd7d6993af0b6763553bf7dcef4cbf980b6b44edfd2f8"}
Oct 30 20:22:46.070658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.070574    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" event={"ID":"27ba866f-d418-4fd2-b23f-de771e91a6f9","Type":"ContainerDied","Data":"6d7500cdaba04156b05daa20ecb354fbd0cfd1c4237e73139ed3286023765231"}
Oct 30 20:22:46.070658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.070611    6058 scope.go:122] "RemoveContainer" containerID="bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a"
Oct 30 20:22:46.072131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.072105    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl" event={"ID":"676c126a-f54a-491d-a9e7-def447e917b0","Type":"ContainerDied","Data":"c275160a56c3a1d9594a3adeb11d9bf247a155217a4f942f314dc450e0feb336"}
Oct 30 20:22:46.072237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.072152    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c275160a56c3a1d9594a3adeb11d9bf247a155217a4f942f314dc450e0feb336"
Oct 30 20:22:46.074127 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.072342    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"
Oct 30 20:22:46.111377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.111341    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:46.121161 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.121137    6058 scope.go:122] "RemoveContainer" containerID="bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a"
Oct 30 20:22:46.121692 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:46.121646    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a\": container with ID starting with bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a not found: ID does not exist" containerID="bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a"
Oct 30 20:22:46.121796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.121691    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a"} err="failed to get container status \"bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a\": rpc error: code = NotFound desc = could not find container \"bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a\": container with ID starting with bc7ad2d1027d7d696c44911700acf43a4beab92de597ddb497719698881e296a not found: ID does not exist"
Oct 30 20:22:46.126099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.126068    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"]
Oct 30 20:22:46.131264 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.131241    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-909519/premium-w2-909519-kserve-7ffd9c4cbc-xt2rb"]
Oct 30 20:22:46.141414 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.141393    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:22:46.222088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.221986    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume\") pod \"27ba866f-d418-4fd2-b23f-de771e91a6f9\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") "
Oct 30 20:22:46.222211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.222191    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb\" (UniqueName: \"kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb\") pod \"27ba866f-d418-4fd2-b23f-de771e91a6f9\" (UID: \"27ba866f-d418-4fd2-b23f-de771e91a6f9\") "
Oct 30 20:22:46.222477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.222450    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume" pod "27ba866f-d418-4fd2-b23f-de771e91a6f9" (UID: "27ba866f-d418-4fd2-b23f-de771e91a6f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:46.222833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.222802    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba866f-d418-4fd2-b23f-de771e91a6f9-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.224396 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.224364    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb" pod "27ba866f-d418-4fd2-b23f-de771e91a6f9" (UID: "27ba866f-d418-4fd2-b23f-de771e91a6f9"). InnerVolumeSpecName "kube-api-access-77vbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.323755 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.323719    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vbb\" (UniqueName: \"kubernetes.io/projected/27ba866f-d418-4fd2-b23f-de771e91a6f9-kube-api-access-77vbb\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.611381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.611349    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:22:46.626916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.626884    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.626945    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.626973    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627063    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627139    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627218    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.627349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627243    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627267    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627306    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627330    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg\") pod \"51195988-aabb-45b5-be2c-332ad185258c\" (UID: \"51195988-aabb-45b5-be2c-332ad185258c\") "
Oct 30 20:22:46.627545 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627405    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.627778 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627754    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.627778 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627772    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.627778 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.627756    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.628213 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.628189    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:46.628306 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.628244    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.630394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.630368    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.632364 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.632339    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:46.632636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.632610    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.632749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.632648    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg" pod "51195988-aabb-45b5-be2c-332ad185258c" (UID: "51195988-aabb-45b5-be2c-332ad185258c"). InnerVolumeSpecName "kube-api-access-jpvhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.670154 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.669812    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:22:46.691045 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.690956    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:46.728219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.728170    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.728453 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.728429    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.728601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.728494    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.728601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.728585    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.728737 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.728713    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.728812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.728795    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.729161 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.729136    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.729272 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.729150    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.729387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.729371    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.729534 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.729517    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.730730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.729640    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.731076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731055    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.731221 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731207    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.731414 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731397    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.731574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731560    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.731676 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731661    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.731799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731784    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.732313 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.731887    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo\") pod \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\" (UID: \"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa\") "
Oct 30 20:22:46.732517 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.732499    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy\") pod \"2210b840-941b-4d46-8777-86aeabd3af6f\" (UID: \"2210b840-941b-4d46-8777-86aeabd3af6f\") "
Oct 30 20:22:46.733471 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733448    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.733583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733570    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/51195988-aabb-45b5-be2c-332ad185258c-kube-api-access-jpvhg\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.733668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733657    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51195988-aabb-45b5-be2c-332ad185258c-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.733744 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733734    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.733833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733821    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.733924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733913    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51195988-aabb-45b5-be2c-332ad185258c-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.734018 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733994    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51195988-aabb-45b5-be2c-332ad185258c-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.734102 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.734090    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.734986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.729384    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.734986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.732064    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.734986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.733546    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.734986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.734688    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.735563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.735411    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:46.736270 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.735993    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:46.739104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.739073    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.739562 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.739090    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.739677 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.739263    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.741179 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.741140    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.744126 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.744089    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "kube-api-access-fphbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.744451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.744428    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.744590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.744567    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:46.747162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.747139    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:46.747254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.747232    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "kube-api-access-kp664". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.748183 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.748160    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token" pod "2210b840-941b-4d46-8777-86aeabd3af6f" (UID: "2210b840-941b-4d46-8777-86aeabd3af6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:46.748759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.748736    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo" pod "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" (UID: "bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835608    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835644    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835659    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835676    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2210b840-941b-4d46-8777-86aeabd3af6f-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835690    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835703    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835716    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2210b840-941b-4d46-8777-86aeabd3af6f-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835730    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835742    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp664\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-kube-api-access-kp664\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835756    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.835763 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835769    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.836393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835781    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.836393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835792    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2210b840-941b-4d46-8777-86aeabd3af6f-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.836393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835804    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2210b840-941b-4d46-8777-86aeabd3af6f-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.836393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835815    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fphbl\" (UniqueName: \"kubernetes.io/projected/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-kube-api-access-fphbl\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.836393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835827    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:46.836393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:46.835840    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:47.084667 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.084624    6058 generic.go:356] "Generic (PLEG): container finished" podID="2210b840-941b-4d46-8777-86aeabd3af6f" containerID="7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037" exitCode=0
Oct 30 20:22:47.084839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.084683    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"
Oct 30 20:22:47.084839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.084675    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" event={"ID":"2210b840-941b-4d46-8777-86aeabd3af6f","Type":"ContainerDied","Data":"7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037"}
Oct 30 20:22:47.084839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.084771    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4" event={"ID":"2210b840-941b-4d46-8777-86aeabd3af6f","Type":"ContainerDied","Data":"a3ee297af2ad6a5480ef1391c44db87cd8ca55a890629c796eb4755a7173eb92"}
Oct 30 20:22:47.084839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.084817    6058 scope.go:122] "RemoveContainer" containerID="7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037"
Oct 30 20:22:47.087597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.087571    6058 generic.go:356] "Generic (PLEG): container finished" podID="51195988-aabb-45b5-be2c-332ad185258c" containerID="864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19" exitCode=0
Oct 30 20:22:47.087731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.087633    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"
Oct 30 20:22:47.087731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.087648    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" event={"ID":"51195988-aabb-45b5-be2c-332ad185258c","Type":"ContainerDied","Data":"864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19"}
Oct 30 20:22:47.087731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.087724    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc" event={"ID":"51195988-aabb-45b5-be2c-332ad185258c","Type":"ContainerDied","Data":"b4aed418d994b0eb1bc541c137fd230f1e198df44e4d8f2ab083703100d666a2"}
Oct 30 20:22:47.089818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.089785    6058 generic.go:356] "Generic (PLEG): container finished" podID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerID="62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664" exitCode=0
Oct 30 20:22:47.090945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.090198    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" event={"ID":"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa","Type":"ContainerDied","Data":"62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664"}
Oct 30 20:22:47.090945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.090247    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b" event={"ID":"bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa","Type":"ContainerDied","Data":"24b03769103f90af776765dadbb033aea0f42fb33b81e1e4a50ec6236f866ab2"}
Oct 30 20:22:47.090945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.090320    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"
Oct 30 20:22:47.101437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.099161    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" event={"ID":"27ba866f-d418-4fd2-b23f-de771e91a6f9","Type":"ContainerDied","Data":"4eed86cef9c171ed0d99b5ed6d895c725f197dd1512777e1a4503c4af22a5e7c"}
Oct 30 20:22:47.101437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.099210    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:22:47.106375 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.106354    6058 scope.go:122] "RemoveContainer" containerID="99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"
Oct 30 20:22:47.110456 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.110426    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.38:50052\" within 1s: context deadline exceeded"
Oct 30 20:22:47.110543 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.110517    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"
Oct 30 20:22:47.110646 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:47.110630    6058 logging.go:55] [core] [Channel #562 SubChannel #563]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: operation was canceled"
Oct 30 20:22:47.122532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.122514    6058 scope.go:122] "RemoveContainer" containerID="7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037"
Oct 30 20:22:47.123080 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:47.122938    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037\": container with ID starting with 7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037 not found: ID does not exist" containerID="7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037"
Oct 30 20:22:47.123080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.122973    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037"} err="failed to get container status \"7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037\": rpc error: code = NotFound desc = could not find container \"7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037\": container with ID starting with 7769dd65b0b891a624906afb0efe7fce0c7a28b94a410ae74a63282e3b79b037 not found: ID does not exist"
Oct 30 20:22:47.123080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.122995    6058 scope.go:122] "RemoveContainer" containerID="99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"
Oct 30 20:22:47.125346 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:47.125304    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8\": container with ID starting with 99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8 not found: ID does not exist" containerID="99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"
Oct 30 20:22:47.125439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.125358    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8"} err="failed to get container status \"99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8\": rpc error: code = NotFound desc = could not find container \"99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8\": container with ID starting with 99426066bc7078b50807aed2f994abf09a5ef36c200b5144e3ae9174cd1be1f8 not found: ID does not exist"
Oct 30 20:22:47.125439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.125385    6058 scope.go:122] "RemoveContainer" containerID="864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19"
Oct 30 20:22:47.135948 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.135931    6058 scope.go:122] "RemoveContainer" containerID="864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19"
Oct 30 20:22:47.136294 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:47.136264    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19\": container with ID starting with 864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19 not found: ID does not exist" containerID="864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19"
Oct 30 20:22:47.136359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.136298    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19"} err="failed to get container status \"864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19\": rpc error: code = NotFound desc = could not find container \"864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19\": container with ID starting with 864a77484444da06cd4e72cb346df9eadf201fcb1deaad8f23ca5b15b6c61d19 not found: ID does not exist"
Oct 30 20:22:47.136359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.136313    6058 scope.go:122] "RemoveContainer" containerID="62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664"
Oct 30 20:22:47.142044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.142022    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"]
Oct 30 20:22:47.146574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.146553    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-4m4z4"]
Oct 30 20:22:47.146637 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.146607    6058 scope.go:122] "RemoveContainer" containerID="62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664"
Oct 30 20:22:47.146908 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:47.146877    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664\": container with ID starting with 62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664 not found: ID does not exist" containerID="62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664"
Oct 30 20:22:47.146982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.146905    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664"} err="failed to get container status \"62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664\": rpc error: code = NotFound desc = could not find container \"62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664\": container with ID starting with 62c2ead26518734d6d633d2640b0034f015b6e12a4ff66d27c634d2924ba9664 not found: ID does not exist"
Oct 30 20:22:47.146982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.146925    6058 scope.go:122] "RemoveContainer" containerID="6d7500cdaba04156b05daa20ecb354fbd0cfd1c4237e73139ed3286023765231"
Oct 30 20:22:47.159932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.159503    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"]
Oct 30 20:22:47.168600 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.168578    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-tqrxc"]
Oct 30 20:22:47.188784 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.188763    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"]
Oct 30 20:22:47.195440 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.195418    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44135b-695f7986d6-h6wfp"]
Oct 30 20:22:47.204838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.204815    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"]
Oct 30 20:22:47.215184 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.215158    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" path="/var/lib/kubelet/pods/2210b840-941b-4d46-8777-86aeabd3af6f/volumes"
Oct 30 20:22:47.215592 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.215576    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" path="/var/lib/kubelet/pods/27ba866f-d418-4fd2-b23f-de771e91a6f9/volumes"
Oct 30 20:22:47.216180 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.216161    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51195988-aabb-45b5-be2c-332ad185258c" path="/var/lib/kubelet/pods/51195988-aabb-45b5-be2c-332ad185258c/volumes"
Oct 30 20:22:47.216952 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.216936    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87374534-095a-4e79-af77-aa9e0717fe49" path="/var/lib/kubelet/pods/87374534-095a-4e79-af77-aa9e0717fe49/volumes"
Oct 30 20:22:47.217359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.217345    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" path="/var/lib/kubelet/pods/9cabd8f3-70d4-404e-9adb-7589b76bb013/volumes"
Oct 30 20:22:47.217639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:47.217627    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c646da-openshift-default-7b65dc6f7c-qsj8b"]
Oct 30 20:22:49.618550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.618510    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" path="/var/lib/kubelet/pods/bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa/volumes"
Oct 30 20:22:49.619612 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.619587    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"]
Oct 30 20:22:49.619819 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.619802    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"]
Oct 30 20:22:49.620023 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.619992    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"]
Oct 30 20:22:49.620372 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.620340    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy" containerID="cri-o://3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447" gracePeriod=30
Oct 30 20:22:49.626050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.620969    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy" containerID="cri-o://86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136" gracePeriod=30
Oct 30 20:22:49.626463 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.626435    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" containerID="cri-o://94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737" gracePeriod=30
Oct 30 20:22:49.816030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.812253    6058 patch_prober.go:29] interesting pod/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:22:49.816030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:49.812325    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:22:50.045995 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:50.045900    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:22:52.442171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:52.442135    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"]
Oct 30 20:22:52.442652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:52.442367    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy" containerID="cri-o://a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12" gracePeriod=30
Oct 30 20:22:52.447219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:52.447189    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"]
Oct 30 20:22:52.447402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:52.447368    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" containerID="cri-o://a1065fabdd060f82d5a1d0b97f78b6174e37d9651c851880b305d61958ebebd2" gracePeriod=30
Oct 30 20:22:52.459201 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:52.459132    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"]
Oct 30 20:22:52.459402 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:52.459372    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy" containerID="cri-o://af07fa15878b19c2fa5ccdf805648d3336b81f98168adabc4f759a6e070faf98" gracePeriod=30
Oct 30 20:22:54.329028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.328395    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"]
Oct 30 20:22:54.329028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.328634    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy" containerID="cri-o://389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889" gracePeriod=30
Oct 30 20:22:54.342088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.336699    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"]
Oct 30 20:22:54.342088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.336903    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy" containerID="cri-o://b6db5f2729ccc9c59140ccd4eb4f26c5166da06bdafb8e48476ec7c1aa79fcc8" gracePeriod=30
Oct 30 20:22:54.695115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.695079    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"]
Oct 30 20:22:54.695927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.695903    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main"
Oct 30 20:22:54.695927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.695927    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.695956    6058 cpu_manager.go:414] "containerMap: removing container" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.695965    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696018    6058 cpu_manager.go:414] "containerMap: removing container" podUID="676c126a-f54a-491d-a9e7-def447e917b0" containerName="revoke-keys"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696027    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676c126a-f54a-491d-a9e7-def447e917b0" containerName="revoke-keys"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696040    6058 cpu_manager.go:414] "containerMap: removing container" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696047    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696057    6058 cpu_manager.go:414] "containerMap: removing container" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696063    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696094    6058 cpu_manager.go:414] "containerMap: removing container" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696102    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696114    6058 cpu_manager.go:414] "containerMap: removing container" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing"
Oct 30 20:22:54.696138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696121    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696243    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="51195988-aabb-45b5-be2c-332ad185258c" containerName="istio-proxy"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696256    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ba866f-d418-4fd2-b23f-de771e91a6f9" containerName="payload-processing"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696268    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cabd8f3-70d4-404e-9adb-7589b76bb013" containerName="main"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696279    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="676c126a-f54a-491d-a9e7-def447e917b0" containerName="revoke-keys"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696290    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696300    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2210b840-941b-4d46-8777-86aeabd3af6f" containerName="istio-proxy"
Oct 30 20:22:54.696732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.696308    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3ac4a9-ff39-449e-8cc9-a58c5ba8d4fa" containerName="istio-proxy"
Oct 30 20:22:54.702815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.702788    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.712500 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.712476    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"]
Oct 30 20:22:54.781216 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781182    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781364 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781262    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781364 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781333    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781368    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781399    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781435    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmtv\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781491    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781551    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.781636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.781586    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883254 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883222    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmtv\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883301    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883368    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883505 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883405    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883505 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883492    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883610 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883571    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883677 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883655    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883702    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.883805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.883736    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.885135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.884518    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.885135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.884533    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.885135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.884778    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.885135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.885099    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.885551 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.885517    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.887419 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.887369    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.887608 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.887584    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.897286 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.897258    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.897392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.897377    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmtv\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv\") pod \"e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") " pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:54.940344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.940325    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:22:54.984815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.984743    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.984815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.984795    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.984913    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.984956    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985080 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985067    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985116    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985163    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985207    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985235    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4\") pod \"98c15f32-cdb3-4063-9335-ecf3e8818d64\" (UID: \"98c15f32-cdb3-4063-9335-ecf3e8818d64\") "
Oct 30 20:22:54.985487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985460    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:54.986507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985969    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:54.986507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.985985    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:54.986507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.986297    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:54.986507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.986423    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:54.986507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.986481    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:54.988114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.988064    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:54.988641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.988608    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "kube-api-access-rxbk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:54.988730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.988688    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:54.990174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:54.990154    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token" pod "98c15f32-cdb3-4063-9335-ecf3e8818d64" (UID: "98c15f32-cdb3-4063-9335-ecf3e8818d64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:55.020849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.020821    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:55.037996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.037972    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"]
Oct 30 20:22:55.042105 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.042084    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81741671-768sp"]
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090504    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090540    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090557    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbk4\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-kube-api-access-rxbk4\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090571    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090588    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c15f32-cdb3-4063-9335-ecf3e8818d64-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090601    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090616    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c15f32-cdb3-4063-9335-ecf3e8818d64-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.090654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.090630    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c15f32-cdb3-4063-9335-ecf3e8818d64-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.213817 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.213744    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487459d8-9003-44e0-b9b6-1c07b58259e9" path="/var/lib/kubelet/pods/487459d8-9003-44e0-b9b6-1c07b58259e9/volumes"
Oct 30 20:22:55.308710 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.308682    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"]
Oct 30 20:22:55.316766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.316401    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:22:55.316766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.316548    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:22:55.316766 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.316618    6058 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:22:55.333634 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:55.333596    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a6c386_8875_43d1_9ea5_439bf99719fa.slice/crio-0f69ba21d4a0ae85980d3a7539f7091d8fc2bdd1659a50973e08589e04812553 WatchSource:0}: Error finding container 0f69ba21d4a0ae85980d3a7539f7091d8fc2bdd1659a50973e08589e04812553: Status 404 returned error can't find the container with id 0f69ba21d4a0ae85980d3a7539f7091d8fc2bdd1659a50973e08589e04812553
Oct 30 20:22:55.503657 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.503634    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:22:55.514552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.514531    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:22:55.544759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.544395    6058 generic.go:356] "Generic (PLEG): container finished" podID="f97445ce-898f-47b9-84d1-df8ee2215258" containerID="94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737" exitCode=0
Oct 30 20:22:55.544759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.544458    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"
Oct 30 20:22:55.544759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.544602    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" event={"ID":"f97445ce-898f-47b9-84d1-df8ee2215258","Type":"ContainerDied","Data":"94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737"}
Oct 30 20:22:55.544759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.544640    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw" event={"ID":"f97445ce-898f-47b9-84d1-df8ee2215258","Type":"ContainerDied","Data":"eae718cf000ce5419de3e97d78d09f4306e3310e8ec2ddee7f1b8afcba9aea7d"}
Oct 30 20:22:55.544759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.544666    6058 scope.go:122] "RemoveContainer" containerID="94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737"
Oct 30 20:22:55.546720 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.546694    6058 generic.go:356] "Generic (PLEG): container finished" podID="9cef02b6-4044-45dc-b462-d829bf44778b" containerID="86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136" exitCode=0
Oct 30 20:22:55.546940 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.546753    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"
Oct 30 20:22:55.547057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.546765    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" event={"ID":"9cef02b6-4044-45dc-b462-d829bf44778b","Type":"ContainerDied","Data":"86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136"}
Oct 30 20:22:55.547134 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.547077    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2" event={"ID":"9cef02b6-4044-45dc-b462-d829bf44778b","Type":"ContainerDied","Data":"e023c33df9a64ea3c9d274ae32a01fd237baf2bee7882f502cb13df96758081b"}
Oct 30 20:22:55.550566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.549552    6058 generic.go:356] "Generic (PLEG): container finished" podID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerID="3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447" exitCode=0
Oct 30 20:22:55.550566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.549655    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" event={"ID":"98c15f32-cdb3-4063-9335-ecf3e8818d64","Type":"ContainerDied","Data":"3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447"}
Oct 30 20:22:55.550566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.549692    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479" event={"ID":"98c15f32-cdb3-4063-9335-ecf3e8818d64","Type":"ContainerDied","Data":"2375f11f17b162deb4d2778a061cbd0fcf6cffd90acf0671c18e22586b042b05"}
Oct 30 20:22:55.550566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.549775    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"
Oct 30 20:22:55.574628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.573701    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" event={"ID":"f9a6c386-8875-43d1-9ea5-439bf99719fa","Type":"ContainerStarted","Data":"e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8"}
Oct 30 20:22:55.574628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.573772    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" event={"ID":"f9a6c386-8875-43d1-9ea5-439bf99719fa","Type":"ContainerStarted","Data":"0f69ba21d4a0ae85980d3a7539f7091d8fc2bdd1659a50973e08589e04812553"}
Oct 30 20:22:55.575328 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.575300    6058 scope.go:122] "RemoveContainer" containerID="94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737"
Oct 30 20:22:55.575686 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:55.575661    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737\": container with ID starting with 94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737 not found: ID does not exist" containerID="94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737"
Oct 30 20:22:55.575783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.575696    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737"} err="failed to get container status \"94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737\": rpc error: code = NotFound desc = could not find container \"94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737\": container with ID starting with 94a3a7f37a5916619fead5cc981453c0813d13b9d0e86570b47ec9ab4b9dc737 not found: ID does not exist"
Oct 30 20:22:55.575783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.575719    6058 scope.go:122] "RemoveContainer" containerID="86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136"
Oct 30 20:22:55.593060 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.593046    6058 scope.go:122] "RemoveContainer" containerID="86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136"
Oct 30 20:22:55.601431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.601220    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"]
Oct 30 20:22:55.608116 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.608089    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-dg479"]
Oct 30 20:22:55.608772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.608748    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.609314 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:55.609117    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136\": container with ID starting with 86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136 not found: ID does not exist" containerID="86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136"
Oct 30 20:22:55.609421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609329    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136"} err="failed to get container status \"86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136\": rpc error: code = NotFound desc = could not find container \"86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136\": container with ID starting with 86137ad74971f27acd89d18175e472e77abe744c7a03c94cda1d5445ea2c2136 not found: ID does not exist"
Oct 30 20:22:55.609421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609353    6058 scope.go:122] "RemoveContainer" containerID="3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447"
Oct 30 20:22:55.609853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609302    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.609853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609645    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.609853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609676    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.609853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609717    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.609853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.609771    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610174    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610223    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610263    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610341    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610395    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610470    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610519    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610563    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610641    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610845    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.611820    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.611871    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.612850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.612106    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.613771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.613278    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:55.613771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.613326    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.613771 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.613359    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.614058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.610687    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.614487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.614235    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data\") pod \"f97445ce-898f-47b9-84d1-df8ee2215258\" (UID: \"f97445ce-898f-47b9-84d1-df8ee2215258\") "
Oct 30 20:22:55.614487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.614451    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.614862 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.614827    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.615026 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.614960    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.615388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.615356    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.615568 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.615513    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs\") pod \"9cef02b6-4044-45dc-b462-d829bf44778b\" (UID: \"9cef02b6-4044-45dc-b462-d829bf44778b\") "
Oct 30 20:22:55.618437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618409    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "kube-api-access-55kjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:55.618591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618572    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618596    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618611    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618625    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618638    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618652    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cef02b6-4044-45dc-b462-d829bf44778b-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618666    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618679    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97445ce-898f-47b9-84d1-df8ee2215258-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618693    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kjp\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-kube-api-access-55kjp\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618708    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618722    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cef02b6-4044-45dc-b462-d829bf44778b-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.618733 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.618736    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.619709 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.619671    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:55.619932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.619896    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "kube-api-access-27js9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:55.620201 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.620166    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:55.620455 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.620424    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo" pod "9cef02b6-4044-45dc-b462-d829bf44778b" (UID: "9cef02b6-4044-45dc-b462-d829bf44778b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:55.620799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.620767    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:55.624469 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.624443    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy" pod "f97445ce-898f-47b9-84d1-df8ee2215258" (UID: "f97445ce-898f-47b9-84d1-df8ee2215258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:55.632693 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.632671    6058 scope.go:122] "RemoveContainer" containerID="3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447"
Oct 30 20:22:55.633056 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:55.633036    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447\": container with ID starting with 3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447 not found: ID does not exist" containerID="3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447"
Oct 30 20:22:55.633134 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.633066    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447"} err="failed to get container status \"3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447\": rpc error: code = NotFound desc = could not find container \"3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447\": container with ID starting with 3e9549000695e8762942252d2a0b6d0fbf3f4ebd369be0fd7c6a59d5301ec447 not found: ID does not exist"
Oct 30 20:22:55.640897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.640856    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podStartSLOduration=1.64084418 podStartE2EDuration="1.64084418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:55.63589824 +0000 UTC m=+1108.539318270" watchObservedRunningTime="2026-10-30 20:22:55.64084418 +0000 UTC m=+1108.544264205"
Oct 30 20:22:55.720490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.720453    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97445ce-898f-47b9-84d1-df8ee2215258-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.720490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.720487    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97445ce-898f-47b9-84d1-df8ee2215258-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.720688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.720500    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97445ce-898f-47b9-84d1-df8ee2215258-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.720688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.720513    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27js9\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-kube-api-access-27js9\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.720688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.720529    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cef02b6-4044-45dc-b462-d829bf44778b-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.720688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.720540    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cef02b6-4044-45dc-b462-d829bf44778b-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:55.875998 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.875970    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"]
Oct 30 20:22:55.883320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.883293    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-n5qzw"]
Oct 30 20:22:55.902421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.902384    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"]
Oct 30 20:22:55.911781 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:55.911746    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-44135b-openshift-default-76f454bf89-twgd2"]
Oct 30 20:22:56.021491 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.021456    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:22:56.022782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.022755    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:22:56.022909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.022803    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:22:56.111078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.110165    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:56.729260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.729227    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"]
Oct 30 20:22:56.729972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.729949    6058 cpu_manager.go:414] "containerMap: removing container" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy"
Oct 30 20:22:56.729972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.729972    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy"
Oct 30 20:22:56.730160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730038    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy"
Oct 30 20:22:56.730160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730048    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy"
Oct 30 20:22:56.730160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730070    6058 cpu_manager.go:414] "containerMap: removing container" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy"
Oct 30 20:22:56.730160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730078    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy"
Oct 30 20:22:56.730341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730217    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" containerName="istio-proxy"
Oct 30 20:22:56.730341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730237    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" containerName="istio-proxy"
Oct 30 20:22:56.730341 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.730251    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" containerName="istio-proxy"
Oct 30 20:22:56.735330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.735306    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.740317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.740295    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:56.741027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.740986    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:56.741132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.741093    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:56.741196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.741164    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:22:56.742327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.742305    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"]
Oct 30 20:22:56.837953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.837916    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.838127 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.837974    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqplt\" (UniqueName: \"kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.838174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.838141    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.939635 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.939600    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.939815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.939699    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.939815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.939736    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqplt\" (UniqueName: \"kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.940390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.940357    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.942017 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.941984    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:56.975030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:56.974873    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqplt\" (UniqueName: \"kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt\") pod \"maas-api-revoke-sub-d903d04e-v8psz\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:57.021950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.021866    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:22:57.022113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.021944    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:22:57.076040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.065445    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"]
Oct 30 20:22:57.085348 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.085316    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04729ff5-sjmdw"]
Oct 30 20:22:57.085522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.085508    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:57.217379 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.217342    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adeadd1-9091-4900-8e85-4b8d0ad98238" path="/var/lib/kubelet/pods/2adeadd1-9091-4900-8e85-4b8d0ad98238/volumes"
Oct 30 20:22:57.218090 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.218070    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c15f32-cdb3-4063-9335-ecf3e8818d64" path="/var/lib/kubelet/pods/98c15f32-cdb3-4063-9335-ecf3e8818d64/volumes"
Oct 30 20:22:57.218688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.218670    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cef02b6-4044-45dc-b462-d829bf44778b" path="/var/lib/kubelet/pods/9cef02b6-4044-45dc-b462-d829bf44778b/volumes"
Oct 30 20:22:57.219754 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.219734    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97445ce-898f-47b9-84d1-df8ee2215258" path="/var/lib/kubelet/pods/f97445ce-898f-47b9-84d1-df8ee2215258/volumes"
Oct 30 20:22:57.375088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.355154    6058 patch_prober.go:29] interesting pod/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:22:57.375088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.355207    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:22:57.383839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.383814    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"]
Oct 30 20:22:57.384283 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:22:57.384258    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60a265c_8d1e_4a2b_8414_aefe15661c18.slice/crio-bb52a498d8761c98e1ba0fa77a0bc244c55ebfac5e9c162f39d84ff335422940 WatchSource:0}: Error finding container bb52a498d8761c98e1ba0fa77a0bc244c55ebfac5e9c162f39d84ff335422940: Status 404 returned error can't find the container with id bb52a498d8761c98e1ba0fa77a0bc244c55ebfac5e9c162f39d84ff335422940
Oct 30 20:22:57.695721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.695505    6058 generic.go:356] "Generic (PLEG): container finished" podID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerID="a1065fabdd060f82d5a1d0b97f78b6174e37d9651c851880b305d61958ebebd2" exitCode=0
Oct 30 20:22:57.695721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.695632    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" event={"ID":"dc843d3e-f474-49d9-8903-101e48aa44ae","Type":"ContainerDied","Data":"a1065fabdd060f82d5a1d0b97f78b6174e37d9651c851880b305d61958ebebd2"}
Oct 30 20:22:57.695721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.695685    6058 scope.go:122] "RemoveContainer" containerID="6201ef598fee2960c8116ef20bde6efc3f2efc93689858841a8a704ab62764d3"
Oct 30 20:22:57.700947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.700440    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz" event={"ID":"b60a265c-8d1e-4a2b-8414-aefe15661c18","Type":"ContainerStarted","Data":"bb52a498d8761c98e1ba0fa77a0bc244c55ebfac5e9c162f39d84ff335422940"}
Oct 30 20:22:57.703607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.703495    6058 generic.go:356] "Generic (PLEG): container finished" podID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerID="af07fa15878b19c2fa5ccdf805648d3336b81f98168adabc4f759a6e070faf98" exitCode=0
Oct 30 20:22:57.703607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:57.703550    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" event={"ID":"2b92260e-31ee-4730-b5aa-88fcedcd6ea0","Type":"ContainerDied","Data":"af07fa15878b19c2fa5ccdf805648d3336b81f98168adabc4f759a6e070faf98"}
Oct 30 20:22:58.034945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.034474    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:22:58.034945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.034558    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:22:58.039251 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.038862    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:22:58.118198 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.117795    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206453    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206514    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206571    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206672    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206737    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206787    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.207136 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.206821    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.207855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.207762    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.208079 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.208037    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.208357 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.208283    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.208545 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.208486    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.209112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.208993    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.209289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.209240    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.209447 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.209402    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.209623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.209562    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo\") pod \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\" (UID: \"2b92260e-31ee-4730-b5aa-88fcedcd6ea0\") "
Oct 30 20:22:58.209878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.209824    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.210097 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.210052    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.210256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.210220    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data\") pod \"dc843d3e-f474-49d9-8903-101e48aa44ae\" (UID: \"dc843d3e-f474-49d9-8903-101e48aa44ae\") "
Oct 30 20:22:58.211639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.211436    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.211940 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.211769    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.212246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.212085    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.212712 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.212379    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.220143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.220113    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.222072 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.220515    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.222260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.221298    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:58.222490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.221527    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.222594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.221812    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:58.223653 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.223621    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.228246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.224805    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "kube-api-access-2n54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:58.228246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.226686    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "kube-api-access-9ggcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:58.228246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.226952    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:58.229106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.229048    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.230788 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.230755    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:58.231074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.231038    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy" pod "dc843d3e-f474-49d9-8903-101e48aa44ae" (UID: "dc843d3e-f474-49d9-8903-101e48aa44ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.231865 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.231842    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:58.232769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.232736    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo" pod "2b92260e-31ee-4730-b5aa-88fcedcd6ea0" (UID: "2b92260e-31ee-4730-b5aa-88fcedcd6ea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:58.234935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.234906    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:22:58.315316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315282    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315312    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315325    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315339    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315353    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315365    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n54r\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-kube-api-access-2n54r\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315378    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315392    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315403    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc843d3e-f474-49d9-8903-101e48aa44ae-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315415    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315427    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315442    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315454    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggcv\" (UniqueName: \"kubernetes.io/projected/dc843d3e-f474-49d9-8903-101e48aa44ae-kube-api-access-9ggcv\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315466    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315478    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b92260e-31ee-4730-b5aa-88fcedcd6ea0-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315492    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315505    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.315561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.315517    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc843d3e-f474-49d9-8903-101e48aa44ae-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.415870 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.415836    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.415929    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.415971    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416094    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416131    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.416208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416166    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416210    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416248    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.416387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416260    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416307    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416364    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy\") pod \"8034d7be-519d-40b7-b608-c452f72a6acf\" (UID: \"8034d7be-519d-40b7-b608-c452f72a6acf\") "
Oct 30 20:22:58.416387 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416379    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.416655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416557    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:58.416796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416768    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.416926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416885    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.416926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416906    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.416926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416920    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.417140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416933    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8034d7be-519d-40b7-b608-c452f72a6acf-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.417140 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.416947    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.418919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.418881    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "kube-api-access-7bxhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:58.419045 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.418971    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:58.419125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.419058    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:58.419401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.419379    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token" pod "8034d7be-519d-40b7-b608-c452f72a6acf" (UID: "8034d7be-519d-40b7-b608-c452f72a6acf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:58.518686 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.518646    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8034d7be-519d-40b7-b608-c452f72a6acf-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.518810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.518692    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.518810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.518707    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8034d7be-519d-40b7-b608-c452f72a6acf-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.518810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.518751    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxhl\" (UniqueName: \"kubernetes.io/projected/8034d7be-519d-40b7-b608-c452f72a6acf-kube-api-access-7bxhl\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:58.713775 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.713733    6058 generic.go:356] "Generic (PLEG): container finished" podID="b60a265c-8d1e-4a2b-8414-aefe15661c18" containerID="d1c06661730d0ebe9ac775c3acaa0570fb84502d9002c8e32d46c48ac3a6932a" exitCode=0
Oct 30 20:22:58.713933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.713821    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz" event={"ID":"b60a265c-8d1e-4a2b-8414-aefe15661c18","Type":"ContainerDied","Data":"d1c06661730d0ebe9ac775c3acaa0570fb84502d9002c8e32d46c48ac3a6932a"}
Oct 30 20:22:58.715388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.715365    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"
Oct 30 20:22:58.715526 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.715388    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng" event={"ID":"2b92260e-31ee-4730-b5aa-88fcedcd6ea0","Type":"ContainerDied","Data":"01e2387b760da938f0a36127b7d1e9322fcadbe13a598514d59c4dd4ecf3a1a5"}
Oct 30 20:22:58.715526 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.715442    6058 scope.go:122] "RemoveContainer" containerID="af07fa15878b19c2fa5ccdf805648d3336b81f98168adabc4f759a6e070faf98"
Oct 30 20:22:58.716881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.716850    6058 generic.go:356] "Generic (PLEG): container finished" podID="8034d7be-519d-40b7-b608-c452f72a6acf" containerID="a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12" exitCode=0
Oct 30 20:22:58.716993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.716912    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"
Oct 30 20:22:58.716993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.716936    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" event={"ID":"8034d7be-519d-40b7-b608-c452f72a6acf","Type":"ContainerDied","Data":"a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12"}
Oct 30 20:22:58.716993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.716989    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx" event={"ID":"8034d7be-519d-40b7-b608-c452f72a6acf","Type":"ContainerDied","Data":"8e47914c88df72d8704fd9861e7ad65bc806b6acae0de92b397c4ff271be5ea3"}
Oct 30 20:22:58.718721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.718667    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc" event={"ID":"dc843d3e-f474-49d9-8903-101e48aa44ae","Type":"ContainerDied","Data":"70eb71f666a0731a0360447369f3ac5d173ef6277faeb44a9f2f833ec5fe8d05"}
Oct 30 20:22:58.718721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.718707    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"
Oct 30 20:22:58.732657 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.732634    6058 scope.go:122] "RemoveContainer" containerID="a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12"
Oct 30 20:22:58.743445 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.743430    6058 scope.go:122] "RemoveContainer" containerID="a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12"
Oct 30 20:22:58.743717 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:58.743695    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12\": container with ID starting with a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12 not found: ID does not exist" containerID="a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12"
Oct 30 20:22:58.743772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.743724    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12"} err="failed to get container status \"a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12\": rpc error: code = NotFound desc = could not find container \"a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12\": container with ID starting with a05283855df8528b2e2b406463d75ebd5f6a99153dae5af5b4b5a4b215eb6c12 not found: ID does not exist"
Oct 30 20:22:58.743772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.743739    6058 scope.go:122] "RemoveContainer" containerID="a1065fabdd060f82d5a1d0b97f78b6174e37d9651c851880b305d61958ebebd2"
Oct 30 20:22:58.783776 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.783737    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"]
Oct 30 20:22:58.788843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.788820    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-9d4qc"]
Oct 30 20:22:58.803831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.803802    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"]
Oct 30 20:22:58.812139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.812111    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-pxbng"]
Oct 30 20:22:58.840890 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.840855    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"]
Oct 30 20:22:58.844330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:58.844306    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-75e734-openshift-default-6d7bc99d4c-q5msx"]
Oct 30 20:22:59.022238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.022163    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:22:59.022238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.022216    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:22:59.214031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.213975    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" path="/var/lib/kubelet/pods/2b92260e-31ee-4730-b5aa-88fcedcd6ea0/volumes"
Oct 30 20:22:59.214569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.214552    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" path="/var/lib/kubelet/pods/8034d7be-519d-40b7-b608-c452f72a6acf/volumes"
Oct 30 20:22:59.215047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.215031    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" path="/var/lib/kubelet/pods/dc843d3e-f474-49d9-8903-101e48aa44ae/volumes"
Oct 30 20:22:59.699061 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.698811    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:22:59.732921 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.732893    6058 generic.go:356] "Generic (PLEG): container finished" podID="a869dc70-8829-44c3-9810-6045910e233b" containerID="b6db5f2729ccc9c59140ccd4eb4f26c5166da06bdafb8e48476ec7c1aa79fcc8" exitCode=0
Oct 30 20:22:59.733050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.732990    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" event={"ID":"a869dc70-8829-44c3-9810-6045910e233b","Type":"ContainerDied","Data":"b6db5f2729ccc9c59140ccd4eb4f26c5166da06bdafb8e48476ec7c1aa79fcc8"}
Oct 30 20:22:59.734845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.734822    6058 generic.go:356] "Generic (PLEG): container finished" podID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerID="389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889" exitCode=0
Oct 30 20:22:59.734921 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.734887    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" event={"ID":"c35a3764-7e24-4911-bffa-65ad0c60cdf3","Type":"ContainerDied","Data":"389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889"}
Oct 30 20:22:59.734921 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.734918    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk" event={"ID":"c35a3764-7e24-4911-bffa-65ad0c60cdf3","Type":"ContainerDied","Data":"b00d727f20994afb5bda8fb14b1c9fca750e0747d77c4324d110c33a4b27acea"}
Oct 30 20:22:59.735017 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.734938    6058 scope.go:122] "RemoveContainer" containerID="389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889"
Oct 30 20:22:59.735105 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.735093    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"
Oct 30 20:22:59.744852 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.744830    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:22:59.747255 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.747231    6058 scope.go:122] "RemoveContainer" containerID="389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889"
Oct 30 20:22:59.747566 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:22:59.747543    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889\": container with ID starting with 389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889 not found: ID does not exist" containerID="389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889"
Oct 30 20:22:59.747609 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.747579    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889"} err="failed to get container status \"389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889\": rpc error: code = NotFound desc = could not find container \"389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889\": container with ID starting with 389cd421aa8ab8bb9b5a95acdd62fd093d35c2244621e1205815baafb1a23889 not found: ID does not exist"
Oct 30 20:22:59.831659 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831615    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.831832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831688    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.831832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831793    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.831984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831841    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.831984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831896    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.831984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831950    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.832162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.831987    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.832162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.832058    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.832162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.832152    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token\") pod \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\" (UID: \"c35a3764-7e24-4911-bffa-65ad0c60cdf3\") "
Oct 30 20:22:59.833833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.833308    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.833833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.833329    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.833833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.833699    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:59.833833 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.833802    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.834156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.833962    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.835496 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.835472    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.835680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.835651    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:59.835680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.835660    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "kube-api-access-lzq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:59.836078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.836055    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo" pod "c35a3764-7e24-4911-bffa-65ad0c60cdf3" (UID: "c35a3764-7e24-4911-bffa-65ad0c60cdf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:59.933073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933042    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933105    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933170    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933259    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933279    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933340 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933324    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933349    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933387    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933422    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy\") pod \"a869dc70-8829-44c3-9810-6045910e233b\" (UID: \"a869dc70-8829-44c3-9810-6045910e233b\") "
Oct 30 20:22:59.933485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933417    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.933485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933429    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.933485 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933470    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:22:59.933850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933538    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.933850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933544    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:59.933850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933644    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933896    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933919    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933935    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933951    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933966    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933980    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a869dc70-8829-44c3-9810-6045910e233b-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.933994    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934020    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934034    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934048    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934063    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzq5r\" (UniqueName: \"kubernetes.io/projected/c35a3764-7e24-4911-bffa-65ad0c60cdf3-kube-api-access-lzq5r\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934077    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934091    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.934546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.934104    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35a3764-7e24-4911-bffa-65ad0c60cdf3-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.935597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.935569    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.935710 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.935696    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:59.935874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.935856    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:59.935937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:22:59.935901    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb" pod "a869dc70-8829-44c3-9810-6045910e233b" (UID: "a869dc70-8829-44c3-9810-6045910e233b"). InnerVolumeSpecName "kube-api-access-lvnxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:00.022018 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.021972    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:23:00.022196 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.022047    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:23:00.034658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.034631    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca\") pod \"b60a265c-8d1e-4a2b-8414-aefe15661c18\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") "
Oct 30 20:23:00.034834 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.034766    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt\" (UniqueName: \"kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt\") pod \"b60a265c-8d1e-4a2b-8414-aefe15661c18\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") "
Oct 30 20:23:00.034834 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.034810    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca\") pod \"b60a265c-8d1e-4a2b-8414-aefe15661c18\" (UID: \"b60a265c-8d1e-4a2b-8414-aefe15661c18\") "
Oct 30 20:23:00.035050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.034993    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca" pod "b60a265c-8d1e-4a2b-8414-aefe15661c18" (UID: "b60a265c-8d1e-4a2b-8414-aefe15661c18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:00.035202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.035184    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.035287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.035204    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.035287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.035215    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a869dc70-8829-44c3-9810-6045910e233b-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.035287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.035223    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvnxb\" (UniqueName: \"kubernetes.io/projected/a869dc70-8829-44c3-9810-6045910e233b-kube-api-access-lvnxb\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.035287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.035232    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a869dc70-8829-44c3-9810-6045910e233b-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.036872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.036844    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt" pod "b60a265c-8d1e-4a2b-8414-aefe15661c18" (UID: "b60a265c-8d1e-4a2b-8414-aefe15661c18"). InnerVolumeSpecName "kube-api-access-rqplt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:00.040475 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.040453    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca" pod "b60a265c-8d1e-4a2b-8414-aefe15661c18" (UID: "b60a265c-8d1e-4a2b-8414-aefe15661c18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:00.093682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.093639    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"]
Oct 30 20:23:00.111467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.111432    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-bzgbk"]
Oct 30 20:23:00.135989 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.135955    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqplt\" (UniqueName: \"kubernetes.io/projected/b60a265c-8d1e-4a2b-8414-aefe15661c18-kube-api-access-rqplt\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.135989 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.135988    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60a265c-8d1e-4a2b-8414-aefe15661c18-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.749156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.749117    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg" event={"ID":"a869dc70-8829-44c3-9810-6045910e233b","Type":"ContainerDied","Data":"4862093f82b6db01b08992f8aeb4f953693ed2b06412c9364c8d434e409bac06"}
Oct 30 20:23:00.749622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.749186    6058 scope.go:122] "RemoveContainer" containerID="b6db5f2729ccc9c59140ccd4eb4f26c5166da06bdafb8e48476ec7c1aa79fcc8"
Oct 30 20:23:00.749622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.749135    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"
Oct 30 20:23:00.790301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.790278    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"
Oct 30 20:23:00.790427 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.790314    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz" event={"ID":"b60a265c-8d1e-4a2b-8414-aefe15661c18","Type":"ContainerDied","Data":"bb52a498d8761c98e1ba0fa77a0bc244c55ebfac5e9c162f39d84ff335422940"}
Oct 30 20:23:00.790427 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.790373    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb52a498d8761c98e1ba0fa77a0bc244c55ebfac5e9c162f39d84ff335422940"
Oct 30 20:23:00.811450 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.811425    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"]
Oct 30 20:23:00.815186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:00.815163    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9c901b-openshift-default-bc4d55965-55rtg"]
Oct 30 20:23:01.021405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:01.021325    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:23:01.021405 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:01.021384    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:23:01.220327 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:01.220291    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a869dc70-8829-44c3-9810-6045910e233b" path="/var/lib/kubelet/pods/a869dc70-8829-44c3-9810-6045910e233b/volumes"
Oct 30 20:23:01.220789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:01.220774    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" path="/var/lib/kubelet/pods/c35a3764-7e24-4911-bffa-65ad0c60cdf3/volumes"
Oct 30 20:23:02.021934 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.021900    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:23:02.022389 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.021989    6058 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:23:02.730665 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.730639    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:23:02.764582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.764545    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume\") pod \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") "
Oct 30 20:23:02.764750 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.764647    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld\" (UniqueName: \"kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld\") pod \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\" (UID: \"3bfd3442-ee08-4689-a722-ec4e41e90ab4\") "
Oct 30 20:23:02.765623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.765598    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume" pod "3bfd3442-ee08-4689-a722-ec4e41e90ab4" (UID: "3bfd3442-ee08-4689-a722-ec4e41e90ab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:02.768073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.767972    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld" pod "3bfd3442-ee08-4689-a722-ec4e41e90ab4" (UID: "3bfd3442-ee08-4689-a722-ec4e41e90ab4"). InnerVolumeSpecName "kube-api-access-j9sld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:02.820396 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.820358    6058 generic.go:356] "Generic (PLEG): container finished" podID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerID="36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61" exitCode=0
Oct 30 20:23:02.820565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.820466    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" event={"ID":"3bfd3442-ee08-4689-a722-ec4e41e90ab4","Type":"ContainerDied","Data":"36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61"}
Oct 30 20:23:02.820565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.820506    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8" event={"ID":"3bfd3442-ee08-4689-a722-ec4e41e90ab4","Type":"ContainerDied","Data":"5398f47f9df0d1674f612e788d22c21d8672ad0fc240845f1afd64a015e940a3"}
Oct 30 20:23:02.820565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.820527    6058 scope.go:122] "RemoveContainer" containerID="36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61"
Oct 30 20:23:02.820740 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.820640    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"
Oct 30 20:23:02.840237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.840214    6058 scope.go:122] "RemoveContainer" containerID="36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61"
Oct 30 20:23:02.840510 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:02.840489    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61\": container with ID starting with 36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61 not found: ID does not exist" containerID="36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61"
Oct 30 20:23:02.840565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.840518    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61"} err="failed to get container status \"36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61\": rpc error: code = NotFound desc = could not find container \"36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61\": container with ID starting with 36b959dd8dee0bf3ddca2280bed8de1216fdbd27a79df5c8f992d93bb2cbdd61 not found: ID does not exist"
Oct 30 20:23:02.848053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.848026    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"]
Oct 30 20:23:02.855298 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.855270    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-909519-777c997d4-z6bz8"]
Oct 30 20:23:02.865745 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.865723    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfd3442-ee08-4689-a722-ec4e41e90ab4-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:02.865845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:02.865746    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9sld\" (UniqueName: \"kubernetes.io/projected/3bfd3442-ee08-4689-a722-ec4e41e90ab4-kube-api-access-j9sld\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:03.025064 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:03.024969    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:23:03.025513 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:03.025393    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:23:03.026067 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:03.026047    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:23:03.203155 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:03.203116    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:23:03.222065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:03.222027    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" path="/var/lib/kubelet/pods/3bfd3442-ee08-4689-a722-ec4e41e90ab4/volumes"
Oct 30 20:23:05.026152 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:05.026118    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"]
Oct 30 20:23:05.029690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:05.029665    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad7a3060-8zgrx"]
Oct 30 20:23:05.214359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:05.214327    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2172cdc-054e-4e01-9670-3b2bacb46b13" path="/var/lib/kubelet/pods/b2172cdc-054e-4e01-9670-3b2bacb46b13/volumes"
Oct 30 20:23:07.027256 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.027223    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"]
Oct 30 20:23:07.030831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.030805    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef31f55-dh6v2"]
Oct 30 20:23:07.212939 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.212905    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe2dfc3a-4fbc-4714-b988-312e75702733" path="/var/lib/kubelet/pods/fe2dfc3a-4fbc-4714-b988-312e75702733/volumes"
Oct 30 20:23:07.576547 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.576504    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"]
Oct 30 20:23:07.577042 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.576969    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy" containerID="cri-o://5f5245684aaea5a6f5b48f2c7ad58df8ff1d9a637b01da6ef5f035e744d9e400" gracePeriod=30
Oct 30 20:23:07.583199 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.583175    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"]
Oct 30 20:23:07.583552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.583523    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy" containerID="cri-o://8c72767e779cfab71d33cc2751582eaaef0cc4df7927e5dd951df91ab93af159" gracePeriod=30
Oct 30 20:23:07.599999 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.599963    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"]
Oct 30 20:23:07.600577 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:07.600408    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy" containerID="cri-o://f1ca5781e504d243a217f96e5c4af2535d9c7141335650a656d7c928efa53b69" gracePeriod=30
Oct 30 20:23:08.033446 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033411    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"]
Oct 30 20:23:08.033878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033861    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033880    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033896    6058 cpu_manager.go:414] "containerMap: removing container" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033904    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033914    6058 cpu_manager.go:414] "containerMap: removing container" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerName="payload-processing"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033920    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerName="payload-processing"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033927    6058 cpu_manager.go:414] "containerMap: removing container" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033932    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033942    6058 cpu_manager.go:414] "containerMap: removing container" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033947    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033959    6058 cpu_manager.go:414] "containerMap: removing container" podUID="b60a265c-8d1e-4a2b-8414-aefe15661c18" containerName="revoke-keys"
Oct 30 20:23:08.033970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033964    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60a265c-8d1e-4a2b-8414-aefe15661c18" containerName="revoke-keys"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033980    6058 cpu_manager.go:414] "containerMap: removing container" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.033988    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034020    6058 cpu_manager.go:414] "containerMap: removing container" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034028    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034146    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034157    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfd3442-ee08-4689-a722-ec4e41e90ab4" containerName="payload-processing"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034165    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35a3764-7e24-4911-bffa-65ad0c60cdf3" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034171    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b92260e-31ee-4730-b5aa-88fcedcd6ea0" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034177    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60a265c-8d1e-4a2b-8414-aefe15661c18" containerName="revoke-keys"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034184    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a869dc70-8829-44c3-9810-6045910e233b" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034190    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="8034d7be-519d-40b7-b608-c452f72a6acf" containerName="istio-proxy"
Oct 30 20:23:08.034368 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.034197    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc843d3e-f474-49d9-8903-101e48aa44ae" containerName="istio-proxy"
Oct 30 20:23:08.039429 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.039408    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.046267 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.046243    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:08.046355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.046297    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:23:08.047779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.047757    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:08.047779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.047774    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:08.054628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.054593    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"]
Oct 30 20:23:08.141377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.141345    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.141520 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.141409    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xx4x\" (UniqueName: \"kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.141595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.141571    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.243163 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.243133    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.243309 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.243262    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.243370 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.243320    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xx4x\" (UniqueName: \"kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.243865 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.243846    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.245504 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.245486    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.257288 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.257261    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xx4x\" (UniqueName: \"kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x\") pod \"maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.356031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.355936    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:08.641454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.641425    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"]
Oct 30 20:23:08.649035 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:23:08.642983    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5363d2eb_3ea3_407d_a07c_b237811216a4.slice/crio-769d99d95e7400dfa22370a50612d35a63f7f9fed544d49e61a24f386b489e78 WatchSource:0}: Error finding container 769d99d95e7400dfa22370a50612d35a63f7f9fed544d49e61a24f386b489e78: Status 404 returned error can't find the container with id 769d99d95e7400dfa22370a50612d35a63f7f9fed544d49e61a24f386b489e78
Oct 30 20:23:08.873099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:08.873062    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8" event={"ID":"5363d2eb-3ea3-407d-a07c-b237811216a4","Type":"ContainerStarted","Data":"769d99d95e7400dfa22370a50612d35a63f7f9fed544d49e61a24f386b489e78"}
Oct 30 20:23:09.028334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:09.028292    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"]
Oct 30 20:23:09.031958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:09.031928    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4cc090-xbbpj"]
Oct 30 20:23:09.216219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:09.216139    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde00931-8b7e-4a35-9513-ada054139ac3" path="/var/lib/kubelet/pods/cde00931-8b7e-4a35-9513-ada054139ac3/volumes"
Oct 30 20:23:09.883882 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:09.883848    6058 generic.go:356] "Generic (PLEG): container finished" podID="5363d2eb-3ea3-407d-a07c-b237811216a4" containerID="856f98045a316ad547acec9c10e9aadc0810994fd31fa98638553b656bb6a353" exitCode=0
Oct 30 20:23:09.884077 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:09.883929    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8" event={"ID":"5363d2eb-3ea3-407d-a07c-b237811216a4","Type":"ContainerDied","Data":"856f98045a316ad547acec9c10e9aadc0810994fd31fa98638553b656bb6a353"}
Oct 30 20:23:10.025828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:10.025792    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"]
Oct 30 20:23:10.029806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:10.029776    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2770f9b-vwwhb"]
Oct 30 20:23:11.038433 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.038411    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:11.076330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.076304    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x\" (UniqueName: \"kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x\") pod \"5363d2eb-3ea3-407d-a07c-b237811216a4\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") "
Oct 30 20:23:11.076498 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.076391    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca\") pod \"5363d2eb-3ea3-407d-a07c-b237811216a4\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") "
Oct 30 20:23:11.076498 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.076470    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca\") pod \"5363d2eb-3ea3-407d-a07c-b237811216a4\" (UID: \"5363d2eb-3ea3-407d-a07c-b237811216a4\") "
Oct 30 20:23:11.077059 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.077031    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca" pod "5363d2eb-3ea3-407d-a07c-b237811216a4" (UID: "5363d2eb-3ea3-407d-a07c-b237811216a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:11.079061 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.078867    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x" pod "5363d2eb-3ea3-407d-a07c-b237811216a4" (UID: "5363d2eb-3ea3-407d-a07c-b237811216a4"). InnerVolumeSpecName "kube-api-access-9xx4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:11.079227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.079089    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca" pod "5363d2eb-3ea3-407d-a07c-b237811216a4" (UID: "5363d2eb-3ea3-407d-a07c-b237811216a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:11.178556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.178466    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xx4x\" (UniqueName: \"kubernetes.io/projected/5363d2eb-3ea3-407d-a07c-b237811216a4-kube-api-access-9xx4x\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:11.178556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.178504    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:11.178556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.178520    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363d2eb-3ea3-407d-a07c-b237811216a4-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:11.212582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.212551    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff69aec-5262-4f87-a45f-25b8ac8988a3" path="/var/lib/kubelet/pods/6ff69aec-5262-4f87-a45f-25b8ac8988a3/volumes"
Oct 30 20:23:11.910581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.910546    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"
Oct 30 20:23:11.910750 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.910581    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8" event={"ID":"5363d2eb-3ea3-407d-a07c-b237811216a4","Type":"ContainerDied","Data":"769d99d95e7400dfa22370a50612d35a63f7f9fed544d49e61a24f386b489e78"}
Oct 30 20:23:11.910750 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:11.910634    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="769d99d95e7400dfa22370a50612d35a63f7f9fed544d49e61a24f386b489e78"
Oct 30 20:23:12.740717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.740669    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"]
Oct 30 20:23:12.929950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.929559    6058 generic.go:356] "Generic (PLEG): container finished" podID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerID="5f5245684aaea5a6f5b48f2c7ad58df8ff1d9a637b01da6ef5f035e744d9e400" exitCode=0
Oct 30 20:23:12.929950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.929860    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" event={"ID":"d84c7cda-57bf-438c-9329-a337f1bdcc96","Type":"ContainerDied","Data":"5f5245684aaea5a6f5b48f2c7ad58df8ff1d9a637b01da6ef5f035e744d9e400"}
Oct 30 20:23:12.932843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.932820    6058 generic.go:356] "Generic (PLEG): container finished" podID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerID="8c72767e779cfab71d33cc2751582eaaef0cc4df7927e5dd951df91ab93af159" exitCode=0
Oct 30 20:23:12.932972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.932907    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" event={"ID":"e4b2a700-f272-4d97-99e7-e894bf75f21f","Type":"ContainerDied","Data":"8c72767e779cfab71d33cc2751582eaaef0cc4df7927e5dd951df91ab93af159"}
Oct 30 20:23:12.932972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.932955    6058 scope.go:122] "RemoveContainer" containerID="cb913287298ce1002a3a7ae7c778c96cfaf34ddd93aac1a12f7d790ea870157f"
Oct 30 20:23:12.936584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.936246    6058 generic.go:356] "Generic (PLEG): container finished" podID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerID="f1ca5781e504d243a217f96e5c4af2535d9c7141335650a656d7c928efa53b69" exitCode=0
Oct 30 20:23:12.936584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:12.936373    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" event={"ID":"37a608f1-c2a1-45a7-a933-e70386b5d7da","Type":"ContainerDied","Data":"f1ca5781e504d243a217f96e5c4af2535d9c7141335650a656d7c928efa53b69"}
Oct 30 20:23:13.054948 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.054902    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"]
Oct 30 20:23:13.064065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.064034    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c882e2-f4mnk"]
Oct 30 20:23:13.075746 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.075716    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:23:13.077851 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.077826    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:23:13.084952 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.084910    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:23:13.216703 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.216605    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f" path="/var/lib/kubelet/pods/b9d8dc50-9f1c-4b3b-9bcc-7327aeb7046f/volumes"
Oct 30 20:23:13.247729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247691    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.247729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247726    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247754    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247790    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247817    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247843    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247867    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247891    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.247927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247908    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247950    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247968    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.247988    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248027    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248055    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248084    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248102    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248122    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248129    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248152    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248209    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248232    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248265    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.248322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248306    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248337    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248134    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248285    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248483    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248506    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248712    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248740    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:13.249029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.248965    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.249058    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:13.249431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.249174    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.249431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.249359    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.250412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.250379    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.250789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.250762    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.250882 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.250857    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "kube-api-access-zfhqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.250967 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.250915    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.251058 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251042    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.251123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251062    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.251123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251075    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert\") pod \"37a608f1-c2a1-45a7-a933-e70386b5d7da\" (UID: \"37a608f1-c2a1-45a7-a933-e70386b5d7da\") "
Oct 30 20:23:13.251123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251078    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.251123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251110    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs\") pod \"e4b2a700-f272-4d97-99e7-e894bf75f21f\" (UID: \"e4b2a700-f272-4d97-99e7-e894bf75f21f\") "
Oct 30 20:23:13.251349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251171    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo\") pod \"d84c7cda-57bf-438c-9329-a337f1bdcc96\" (UID: \"d84c7cda-57bf-438c-9329-a337f1bdcc96\") "
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251498    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "kube-api-access-v8j55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251564    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251581    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251597    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhqd\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-kube-api-access-zfhqd\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251611    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251623    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251636    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251649    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251661    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251674    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251687    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251700    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251712    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b2a700-f272-4d97-99e7-e894bf75f21f-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251725    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84c7cda-57bf-438c-9329-a337f1bdcc96-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251739    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251751    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251763    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251774    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84c7cda-57bf-438c-9329-a337f1bdcc96-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.252220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.251836    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:13.260527 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.260501    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.260688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.260549    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:13.260950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.260554    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.260950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.260865    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k" pod "d84c7cda-57bf-438c-9329-a337f1bdcc96" (UID: "d84c7cda-57bf-438c-9329-a337f1bdcc96"). InnerVolumeSpecName "kube-api-access-xfc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.260950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.260913    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.261170 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.261126    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:13.261560 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.261540    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo" pod "37a608f1-c2a1-45a7-a933-e70386b5d7da" (UID: "37a608f1-c2a1-45a7-a933-e70386b5d7da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:13.288120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.288078    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs" pod "e4b2a700-f272-4d97-99e7-e894bf75f21f" (UID: "e4b2a700-f272-4d97-99e7-e894bf75f21f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:13.353219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353173    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353215    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353231    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353246    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a608f1-c2a1-45a7-a933-e70386b5d7da-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353261    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b2a700-f272-4d97-99e7-e894bf75f21f-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353275    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a608f1-c2a1-45a7-a933-e70386b5d7da-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353291    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b2a700-f272-4d97-99e7-e894bf75f21f-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353307    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84c7cda-57bf-438c-9329-a337f1bdcc96-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353322    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8j55\" (UniqueName: \"kubernetes.io/projected/37a608f1-c2a1-45a7-a933-e70386b5d7da-kube-api-access-v8j55\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.353390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.353339    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfc7k\" (UniqueName: \"kubernetes.io/projected/d84c7cda-57bf-438c-9329-a337f1bdcc96-kube-api-access-xfc7k\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:13.945202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.945111    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"
Oct 30 20:23:13.945202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.945130    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk" event={"ID":"d84c7cda-57bf-438c-9329-a337f1bdcc96","Type":"ContainerDied","Data":"5d5c8ee1a09d761ca8baa5ec810472b1984bb52514b80e23fc057a66c5feef40"}
Oct 30 20:23:13.945665 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.945208    6058 scope.go:122] "RemoveContainer" containerID="5f5245684aaea5a6f5b48f2c7ad58df8ff1d9a637b01da6ef5f035e744d9e400"
Oct 30 20:23:13.946872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.946855    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"
Oct 30 20:23:13.946997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.946852    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p" event={"ID":"e4b2a700-f272-4d97-99e7-e894bf75f21f","Type":"ContainerDied","Data":"521bfc5d95dd5d779f93f7c10bb00278a34883a975e642bf3fba4191cc2c00ea"}
Oct 30 20:23:13.948483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.948452    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf" event={"ID":"37a608f1-c2a1-45a7-a933-e70386b5d7da","Type":"ContainerDied","Data":"3915041624aaf9cd2370e8c153f98645502b6b5ca8a133337f98d0c9bbf26e2c"}
Oct 30 20:23:13.948599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.948457    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"
Oct 30 20:23:13.959027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.958977    6058 scope.go:122] "RemoveContainer" containerID="8c72767e779cfab71d33cc2751582eaaef0cc4df7927e5dd951df91ab93af159"
Oct 30 20:23:13.972214 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.972189    6058 scope.go:122] "RemoveContainer" containerID="f1ca5781e504d243a217f96e5c4af2535d9c7141335650a656d7c928efa53b69"
Oct 30 20:23:13.979185 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.979154    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"]
Oct 30 20:23:13.985451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.985429    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-7f6bk"]
Oct 30 20:23:13.999769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:13.999746    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"]
Oct 30 20:23:14.010782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:14.010760    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-jgxdf"]
Oct 30 20:23:14.024289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:14.024267    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"]
Oct 30 20:23:14.028913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:14.028893    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-909519-openshift-default-665577cd95-kbm8p"]
Oct 30 20:23:15.211582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:15.211549    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" path="/var/lib/kubelet/pods/37a608f1-c2a1-45a7-a933-e70386b5d7da/volumes"
Oct 30 20:23:15.212019 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:15.211993    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" path="/var/lib/kubelet/pods/d84c7cda-57bf-438c-9329-a337f1bdcc96/volumes"
Oct 30 20:23:15.212413 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:15.212399    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" path="/var/lib/kubelet/pods/e4b2a700-f272-4d97-99e7-e894bf75f21f/volumes"
Oct 30 20:23:18.143386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.143352    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"]
Oct 30 20:23:18.681569 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.681534    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"]
Oct 30 20:23:18.682076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682057    6058 cpu_manager.go:414] "containerMap: removing container" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy"
Oct 30 20:23:18.682076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682075    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy"
Oct 30 20:23:18.682076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682097    6058 cpu_manager.go:414] "containerMap: removing container" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682104    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682125    6058 cpu_manager.go:414] "containerMap: removing container" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682132    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682150    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5363d2eb-3ea3-407d-a07c-b237811216a4" containerName="revoke-keys"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682158    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5363d2eb-3ea3-407d-a07c-b237811216a4" containerName="revoke-keys"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682288    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a608f1-c2a1-45a7-a933-e70386b5d7da" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682307    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682320    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84c7cda-57bf-438c-9329-a337f1bdcc96" containerName="istio-proxy"
Oct 30 20:23:18.682322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682326    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy"
Oct 30 20:23:18.682767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.682335    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5363d2eb-3ea3-407d-a07c-b237811216a4" containerName="revoke-keys"
Oct 30 20:23:18.687746 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.687720    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.692988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.692961    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:18.693139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.693075    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:18.693451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.693428    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"]
Oct 30 20:23:18.735278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.735242    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:18.735432 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.735344    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:23:18.760377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.760337    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:23:18.811418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.811380    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxmd\" (UniqueName: \"kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.811418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.811420    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.811645 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.811586    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.912646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.912599    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.912849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.912748    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxmd\" (UniqueName: \"kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.912849 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.912786    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.913573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.913541    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.915642 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.915616    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:18.934104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:18.934023    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxmd\" (UniqueName: \"kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd\") pod \"maas-api-revoke-sub-31e5915f-tbc2f\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:19.005307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:19.005273    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:19.343096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:19.342113    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"]
Oct 30 20:23:19.679298 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:19.679257    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:23:20.012690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:20.012595    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f" event={"ID":"2e4e032a-d5c4-418c-97a2-55518640ada6","Type":"ContainerStarted","Data":"d33133d9ffaa2e913024720364adc8660b25c8ec190ed7dfa8eea2f4098bc2ca"}
Oct 30 20:23:20.012690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:20.012670    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f" event={"ID":"2e4e032a-d5c4-418c-97a2-55518640ada6","Type":"ContainerStarted","Data":"39aa68fccc27eb7aa997866d6e2640757864fa14c0b31fc33a1e01f41fa64d92"}
Oct 30 20:23:20.026541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:20.026510    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"]
Oct 30 20:23:20.033488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:20.033450    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd7bc4d-pc5vf"]
Oct 30 20:23:20.037571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:20.037524    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f" podStartSLOduration=2.037511085 podStartE2EDuration="2.037511085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:20.033359189 +0000 UTC m=+1132.936779214" watchObservedRunningTime="2026-10-30 20:23:20.037511085 +0000 UTC m=+1132.940931111"
Oct 30 20:23:21.021835 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:21.021801    6058 generic.go:356] "Generic (PLEG): container finished" podID="2e4e032a-d5c4-418c-97a2-55518640ada6" containerID="d33133d9ffaa2e913024720364adc8660b25c8ec190ed7dfa8eea2f4098bc2ca" exitCode=0
Oct 30 20:23:21.022240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:21.021887    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f" event={"ID":"2e4e032a-d5c4-418c-97a2-55518640ada6","Type":"ContainerDied","Data":"d33133d9ffaa2e913024720364adc8660b25c8ec190ed7dfa8eea2f4098bc2ca"}
Oct 30 20:23:21.026537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:21.026510    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"]
Oct 30 20:23:21.029928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:21.029906    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55b52bf-bn5vs"]
Oct 30 20:23:21.212679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:21.212649    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582fd540-84d2-4fe7-aa89-e8e5212710c8" path="/var/lib/kubelet/pods/582fd540-84d2-4fe7-aa89-e8e5212710c8/volumes"
Oct 30 20:23:21.213099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:21.213085    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad047721-5c69-4d66-b297-6afe97b8b803" path="/var/lib/kubelet/pods/ad047721-5c69-4d66-b297-6afe97b8b803/volumes"
Oct 30 20:23:22.165390 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.165367    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:22.268243 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.268217    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd\" (UniqueName: \"kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd\") pod \"2e4e032a-d5c4-418c-97a2-55518640ada6\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") "
Oct 30 20:23:22.268382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.268274    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca\") pod \"2e4e032a-d5c4-418c-97a2-55518640ada6\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") "
Oct 30 20:23:22.268382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.268349    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca\") pod \"2e4e032a-d5c4-418c-97a2-55518640ada6\" (UID: \"2e4e032a-d5c4-418c-97a2-55518640ada6\") "
Oct 30 20:23:22.268730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.268705    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca" pod "2e4e032a-d5c4-418c-97a2-55518640ada6" (UID: "2e4e032a-d5c4-418c-97a2-55518640ada6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:22.270331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.270312    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca" pod "2e4e032a-d5c4-418c-97a2-55518640ada6" (UID: "2e4e032a-d5c4-418c-97a2-55518640ada6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:22.270331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.270320    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd" pod "2e4e032a-d5c4-418c-97a2-55518640ada6" (UID: "2e4e032a-d5c4-418c-97a2-55518640ada6"). InnerVolumeSpecName "kube-api-access-9vxmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:22.369193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.369167    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vxmd\" (UniqueName: \"kubernetes.io/projected/2e4e032a-d5c4-418c-97a2-55518640ada6-kube-api-access-9vxmd\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:22.369193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.369190    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:22.369318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:22.369200    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e032a-d5c4-418c-97a2-55518640ada6-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.040816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:23.040779    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f" event={"ID":"2e4e032a-d5c4-418c-97a2-55518640ada6","Type":"ContainerDied","Data":"39aa68fccc27eb7aa997866d6e2640757864fa14c0b31fc33a1e01f41fa64d92"}
Oct 30 20:23:23.040979 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:23.040830    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39aa68fccc27eb7aa997866d6e2640757864fa14c0b31fc33a1e01f41fa64d92"
Oct 30 20:23:23.040979 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:23.040833    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"
Oct 30 20:23:23.143810 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:23.143767    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerName="payload-pre-processing" containerID="cri-o://072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061" gracePeriod=25
Oct 30 20:23:26.107607 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:26.107574    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:27.773469 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:27.773418    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main" containerID="cri-o://e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126" gracePeriod=45
Oct 30 20:23:28.029965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.029936    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:23:28.083811 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.083781    6058 generic.go:356] "Generic (PLEG): container finished" podID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerID="e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126" exitCode=0
Oct 30 20:23:28.083988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.083844    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" event={"ID":"35174ada-bc58-4697-83cb-2d0a4db35dfb","Type":"ContainerDied","Data":"e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126"}
Oct 30 20:23:28.083988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.083877    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t" event={"ID":"35174ada-bc58-4697-83cb-2d0a4db35dfb","Type":"ContainerDied","Data":"d3acc4d376f76c1c364491bf64f138e792d75dc3f2eaca3e5e7154bf890869d3"}
Oct 30 20:23:28.083988 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.083902    6058 scope.go:122] "RemoveContainer" containerID="e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126"
Oct 30 20:23:28.084215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.084102    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"
Oct 30 20:23:28.095096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.095063    6058 scope.go:122] "RemoveContainer" containerID="e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126"
Oct 30 20:23:28.095431 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:28.095403    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126\": container with ID starting with e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126 not found: ID does not exist" containerID="e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126"
Oct 30 20:23:28.095488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.095447    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126"} err="failed to get container status \"e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126\": rpc error: code = NotFound desc = could not find container \"e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126\": container with ID starting with e36b50c9779e7c9bdc93616aff5a8707ac6b697edc1071062b21162cdd11b126 not found: ID does not exist"
Oct 30 20:23:28.131527 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131499    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home\") pod \"35174ada-bc58-4697-83cb-2d0a4db35dfb\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") "
Oct 30 20:23:28.131674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131622    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir\") pod \"35174ada-bc58-4697-83cb-2d0a4db35dfb\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") "
Oct 30 20:23:28.131674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131644    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache\") pod \"35174ada-bc58-4697-83cb-2d0a4db35dfb\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") "
Oct 30 20:23:28.131799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131740    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs\") pod \"35174ada-bc58-4697-83cb-2d0a4db35dfb\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") "
Oct 30 20:23:28.131799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131776    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm\") pod \"35174ada-bc58-4697-83cb-2d0a4db35dfb\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") "
Oct 30 20:23:28.131912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131793    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home" pod "35174ada-bc58-4697-83cb-2d0a4db35dfb" (UID: "35174ada-bc58-4697-83cb-2d0a4db35dfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:28.131912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131818    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql\" (UniqueName: \"kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql\") pod \"35174ada-bc58-4697-83cb-2d0a4db35dfb\" (UID: \"35174ada-bc58-4697-83cb-2d0a4db35dfb\") "
Oct 30 20:23:28.132044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.131944    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache" pod "35174ada-bc58-4697-83cb-2d0a4db35dfb" (UID: "35174ada-bc58-4697-83cb-2d0a4db35dfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:28.132322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.132248    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:28.132322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.132275    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir" pod "35174ada-bc58-4697-83cb-2d0a4db35dfb" (UID: "35174ada-bc58-4697-83cb-2d0a4db35dfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:28.132322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.132290    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:28.134250 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.134212    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs" pod "35174ada-bc58-4697-83cb-2d0a4db35dfb" (UID: "35174ada-bc58-4697-83cb-2d0a4db35dfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:28.134467 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.134443    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql" pod "35174ada-bc58-4697-83cb-2d0a4db35dfb" (UID: "35174ada-bc58-4697-83cb-2d0a4db35dfb"). InnerVolumeSpecName "kube-api-access-pwjql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:28.134554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.134518    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm" pod "35174ada-bc58-4697-83cb-2d0a4db35dfb" (UID: "35174ada-bc58-4697-83cb-2d0a4db35dfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:28.232971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.232934    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:28.232971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.232968    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35174ada-bc58-4697-83cb-2d0a4db35dfb-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:28.233229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.232980    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35174ada-bc58-4697-83cb-2d0a4db35dfb-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:28.233229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.232997    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjql\" (UniqueName: \"kubernetes.io/projected/35174ada-bc58-4697-83cb-2d0a4db35dfb-kube-api-access-pwjql\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:28.415687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.415656    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"]
Oct 30 20:23:28.420167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:28.420142    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5197ec/premium-w1-5197ec-kserve-6fd54756f9-gxx7t"]
Oct 30 20:23:29.220171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:29.220137    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" path="/var/lib/kubelet/pods/35174ada-bc58-4697-83cb-2d0a4db35dfb/volumes"
Oct 30 20:23:31.043336 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:31.043291    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"]
Oct 30 20:23:31.047131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:31.047109    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1600ffe8-xxwxs"]
Oct 30 20:23:31.211461 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:31.211425    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ec849f-8646-4935-ba77-737a26790c78" path="/var/lib/kubelet/pods/15ec849f-8646-4935-ba77-737a26790c78/volumes"
Oct 30 20:23:31.632765 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:31.632729    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:23:32.608040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:32.606496    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"]
Oct 30 20:23:32.612272 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:32.612240    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5197ec-bn9k8"]
Oct 30 20:23:33.228699 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:33.228659    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5363d2eb-3ea3-407d-a07c-b237811216a4" path="/var/lib/kubelet/pods/5363d2eb-3ea3-407d-a07c-b237811216a4/volumes"
Oct 30 20:23:36.107023 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:36.106978    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:39.242159 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:39.242123    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:23:41.669050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:41.667648    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"]
Oct 30 20:23:41.669050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:41.667883    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy" containerID="cri-o://7a166363d7cb470dbd8f727718d085d97170d7d86c4cbe4b932d298fca8ed465" gracePeriod=30
Oct 30 20:23:41.679579 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:41.679548    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"]
Oct 30 20:23:41.679885 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:41.679819    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy" containerID="cri-o://e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8" gracePeriod=30
Oct 30 20:23:41.712731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:41.712699    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"]
Oct 30 20:23:41.713151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:41.713118    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" containerID="cri-o://573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303" gracePeriod=30
Oct 30 20:23:43.407092 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.406998    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:23:43.527910 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.527866    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4\" (UniqueName: \"kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4\") pod \"5eb6e6b6-7442-4a01-97f2-8e967b489849\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") "
Oct 30 20:23:43.528089 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.528034    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume\") pod \"5eb6e6b6-7442-4a01-97f2-8e967b489849\" (UID: \"5eb6e6b6-7442-4a01-97f2-8e967b489849\") "
Oct 30 20:23:43.528394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.528367    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume" pod "5eb6e6b6-7442-4a01-97f2-8e967b489849" (UID: "5eb6e6b6-7442-4a01-97f2-8e967b489849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:43.529969 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.529943    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4" pod "5eb6e6b6-7442-4a01-97f2-8e967b489849" (UID: "5eb6e6b6-7442-4a01-97f2-8e967b489849"). InnerVolumeSpecName "kube-api-access-vcpc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:43.629106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.629079    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcpc4\" (UniqueName: \"kubernetes.io/projected/5eb6e6b6-7442-4a01-97f2-8e967b489849-kube-api-access-vcpc4\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:43.629106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:43.629103    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb6e6b6-7442-4a01-97f2-8e967b489849-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:44.227641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.227603    6058 generic.go:356] "Generic (PLEG): container finished" podID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerID="072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061" exitCode=0
Oct 30 20:23:44.227813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.227670    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"
Oct 30 20:23:44.227813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.227731    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" event={"ID":"5eb6e6b6-7442-4a01-97f2-8e967b489849","Type":"ContainerDied","Data":"072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061"}
Oct 30 20:23:44.227813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.227788    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps" event={"ID":"5eb6e6b6-7442-4a01-97f2-8e967b489849","Type":"ContainerDied","Data":"cffb0febf45d08e7df9e256135f932ebe7a3639c097e59edef534c726931f83d"}
Oct 30 20:23:44.227813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.227810    6058 scope.go:122] "RemoveContainer" containerID="072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061"
Oct 30 20:23:44.240465 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.240439    6058 scope.go:122] "RemoveContainer" containerID="072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061"
Oct 30 20:23:44.240952 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:44.240797    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061\": container with ID starting with 072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061 not found: ID does not exist" containerID="072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061"
Oct 30 20:23:44.240952 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.240842    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061"} err="failed to get container status \"072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061\": rpc error: code = NotFound desc = could not find container \"072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061\": container with ID starting with 072f3967017bfa20ba43598af17d378e7e3b5f83469250c30736bf0d9cd49061 not found: ID does not exist"
Oct 30 20:23:44.265324 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.265294    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"]
Oct 30 20:23:44.272674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:44.272647    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5197ec-858489c95-vz8ps"]
Oct 30 20:23:45.027406 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:45.027365    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"]
Oct 30 20:23:45.032572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:45.032539    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87cd34e-2n5dk"]
Oct 30 20:23:45.211702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:45.211666    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" path="/var/lib/kubelet/pods/5eb6e6b6-7442-4a01-97f2-8e967b489849/volumes"
Oct 30 20:23:45.212138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:45.212121    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fdeb58-8c26-4e30-abdd-1ae63affb174" path="/var/lib/kubelet/pods/b9fdeb58-8c26-4e30-abdd-1ae63affb174/volumes"
Oct 30 20:23:46.563740 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:46.563702    6058 patch_prober.go:29] interesting pod/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:23:46.564117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:46.563774    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:23:46.981056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:46.981030    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:23:46.998796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:46.998769    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:23:47.061958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.061923    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062044    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062112    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062185    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062185    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.062240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062227    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062260    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062314    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062349    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062400    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.062533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062411    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy\") pod \"f9a6c386-8875-43d1-9ea5-439bf99719fa\" (UID: \"f9a6c386-8875-43d1-9ea5-439bf99719fa\") "
Oct 30 20:23:47.062533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062443    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.062804 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062779    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:47.062958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062934    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.062958 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062951    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.063129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062962    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.063129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062976    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.063129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.062990    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a6c386-8875-43d1-9ea5-439bf99719fa-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.064606 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.064582    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:47.064933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.064907    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.065055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.064917    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "kube-api-access-bfmtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:47.065055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.064973    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo" pod "f9a6c386-8875-43d1-9ea5-439bf99719fa" (UID: "f9a6c386-8875-43d1-9ea5-439bf99719fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:47.163706 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163680    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.163879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163739    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.163879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163766    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.163879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163784    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.163879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163832    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.164113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163890    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.164113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163963    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.164113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.163998    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.164113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164060    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs\") pod \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\" (UID: \"dbee7645-28fd-4fea-b29a-3956a82e7a8b\") "
Oct 30 20:23:47.164318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164131    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.164318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164175    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.164318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164195    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.164318 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164220    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:47.164562 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164411    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.164618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164573    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164591    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164618 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164607    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164621    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmtv\" (UniqueName: \"kubernetes.io/projected/f9a6c386-8875-43d1-9ea5-439bf99719fa-kube-api-access-bfmtv\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164636    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164650    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164663    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164677    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a6c386-8875-43d1-9ea5-439bf99719fa-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164691    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.164764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.164705    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.166223 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.166200    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "kube-api-access-wzcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:47.166313 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.166226    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:47.166373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.166357    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:47.166407 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.166359    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy" pod "dbee7645-28fd-4fea-b29a-3956a82e7a8b" (UID: "dbee7645-28fd-4fea-b29a-3956a82e7a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.255043 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.254982    6058 generic.go:356] "Generic (PLEG): container finished" podID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerID="7a166363d7cb470dbd8f727718d085d97170d7d86c4cbe4b932d298fca8ed465" exitCode=0
Oct 30 20:23:47.255170 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.255125    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" event={"ID":"f7bacd98-f21a-4ee5-b0ae-caab7dff3218","Type":"ContainerDied","Data":"7a166363d7cb470dbd8f727718d085d97170d7d86c4cbe4b932d298fca8ed465"}
Oct 30 20:23:47.256896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.256861    6058 generic.go:356] "Generic (PLEG): container finished" podID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerID="573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303" exitCode=0
Oct 30 20:23:47.257044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.256904    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"
Oct 30 20:23:47.257044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.256939    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" event={"ID":"dbee7645-28fd-4fea-b29a-3956a82e7a8b","Type":"ContainerDied","Data":"573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303"}
Oct 30 20:23:47.257044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.256999    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p" event={"ID":"dbee7645-28fd-4fea-b29a-3956a82e7a8b","Type":"ContainerDied","Data":"bbf4720849fd9a6d42808dc039bd9499ff574e832bf992a8c8b660c1535231a4"}
Oct 30 20:23:47.257200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.257045    6058 scope.go:122] "RemoveContainer" containerID="573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303"
Oct 30 20:23:47.258858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.258834    6058 generic.go:356] "Generic (PLEG): container finished" podID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerID="e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8" exitCode=0
Oct 30 20:23:47.258971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.258885    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"
Oct 30 20:23:47.258971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.258910    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" event={"ID":"f9a6c386-8875-43d1-9ea5-439bf99719fa","Type":"ContainerDied","Data":"e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8"}
Oct 30 20:23:47.258971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.258950    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74" event={"ID":"f9a6c386-8875-43d1-9ea5-439bf99719fa","Type":"ContainerDied","Data":"0f69ba21d4a0ae85980d3a7539f7091d8fc2bdd1659a50973e08589e04812553"}
Oct 30 20:23:47.265709 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.265685    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.265828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.265716    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.265828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.265733    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee7645-28fd-4fea-b29a-3956a82e7a8b-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.265828 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.265747    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcdk\" (UniqueName: \"kubernetes.io/projected/dbee7645-28fd-4fea-b29a-3956a82e7a8b-kube-api-access-wzcdk\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.271125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.271099    6058 scope.go:122] "RemoveContainer" containerID="573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303"
Oct 30 20:23:47.271465 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:47.271444    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303\": container with ID starting with 573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303 not found: ID does not exist" containerID="573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303"
Oct 30 20:23:47.271559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.271475    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303"} err="failed to get container status \"573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303\": rpc error: code = NotFound desc = could not find container \"573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303\": container with ID starting with 573fc9c9a440c3c3711e36a8c12235fdb82a56404dda955a9a012dfbc79cb303 not found: ID does not exist"
Oct 30 20:23:47.271559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.271497    6058 scope.go:122] "RemoveContainer" containerID="e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8"
Oct 30 20:23:47.281881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.281858    6058 scope.go:122] "RemoveContainer" containerID="e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8"
Oct 30 20:23:47.282219 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:47.282201    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8\": container with ID starting with e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8 not found: ID does not exist" containerID="e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8"
Oct 30 20:23:47.282289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.282227    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8"} err="failed to get container status \"e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8\": rpc error: code = NotFound desc = could not find container \"e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8\": container with ID starting with e68872137084900761b2441a844f87cdb1ebd03d6c9dc8d27a2519e34f592fa8 not found: ID does not exist"
Oct 30 20:23:47.294025 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.293977    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"]
Oct 30 20:23:47.300426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.300382    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-d7p74"]
Oct 30 20:23:47.316692 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.316670    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"]
Oct 30 20:23:47.322587 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.322567    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:23:47.324084 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.324064    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-65f8p"]
Oct 30 20:23:47.468150 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468070    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468150 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468111    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468187    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468211    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468264    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468363 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468308    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468366    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.468641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468387    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468433    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468484    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket\") pod \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\" (UID: \"f7bacd98-f21a-4ee5-b0ae-caab7dff3218\") "
Oct 30 20:23:47.468641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468544    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.468641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468563    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.468967 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468853    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.468967 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468874    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.468967 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.468889    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.469498 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.469471    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:47.469752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.469725    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.470713 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.470689    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:47.470932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.470914    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:47.471040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.470935    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "kube-api-access-lzdjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:47.471276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.471260    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy" pod "f7bacd98-f21a-4ee5-b0ae-caab7dff3218" (UID: "f7bacd98-f21a-4ee5-b0ae-caab7dff3218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:47.569473 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.569443    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.569473 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.569475    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.569860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.569486    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdjb\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-kube-api-access-lzdjb\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.569860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.569495    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.569860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.569507    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:47.569860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:47.569519    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bacd98-f21a-4ee5-b0ae-caab7dff3218-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:48.267068 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:48.267038    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"
Oct 30 20:23:48.267303 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:48.267037    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp" event={"ID":"f7bacd98-f21a-4ee5-b0ae-caab7dff3218","Type":"ContainerDied","Data":"3b20984431cdcada18e382dd4b8c55c43db42e5e04c95670539e9c6196e6c014"}
Oct 30 20:23:48.267420 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:48.267347    6058 scope.go:122] "RemoveContainer" containerID="7a166363d7cb470dbd8f727718d085d97170d7d86c4cbe4b932d298fca8ed465"
Oct 30 20:23:48.293463 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:48.293438    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"]
Oct 30 20:23:48.300168 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:48.300147    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5197ec-openshift-default-7b9b67ff59-thfzp"]
Oct 30 20:23:49.212144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:49.212115    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" path="/var/lib/kubelet/pods/dbee7645-28fd-4fea-b29a-3956a82e7a8b/volumes"
Oct 30 20:23:49.212546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:49.212532    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" path="/var/lib/kubelet/pods/f7bacd98-f21a-4ee5-b0ae-caab7dff3218/volumes"
Oct 30 20:23:49.212914 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:49.212903    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" path="/var/lib/kubelet/pods/f9a6c386-8875-43d1-9ea5-439bf99719fa/volumes"
Oct 30 20:23:49.655651 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:23:49.655615    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:23:51.094588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.094555    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"]
Oct 30 20:23:51.095176 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095155    6058 cpu_manager.go:414] "containerMap: removing container" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095176    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095225    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerName="payload-pre-processing"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095234    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerName="payload-pre-processing"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095252    6058 cpu_manager.go:414] "containerMap: removing container" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095260    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095268    6058 cpu_manager.go:414] "containerMap: removing container" podUID="2e4e032a-d5c4-418c-97a2-55518640ada6" containerName="revoke-keys"
Oct 30 20:23:51.095289 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095276    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4e032a-d5c4-418c-97a2-55518640ada6" containerName="revoke-keys"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095295    6058 cpu_manager.go:414] "containerMap: removing container" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095303    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b2a700-f272-4d97-99e7-e894bf75f21f" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095315    6058 cpu_manager.go:414] "containerMap: removing container" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095343    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095359    6058 cpu_manager.go:414] "containerMap: removing container" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095367    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095482    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bacd98-f21a-4ee5-b0ae-caab7dff3218" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095497    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a6c386-8875-43d1-9ea5-439bf99719fa" containerName="istio-proxy"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095507    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb6e6b6-7442-4a01-97f2-8e967b489849" containerName="payload-pre-processing"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095519    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="35174ada-bc58-4697-83cb-2d0a4db35dfb" containerName="main"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095530    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4e032a-d5c4-418c-97a2-55518640ada6" containerName="revoke-keys"
Oct 30 20:23:51.095627 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.095542    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbee7645-28fd-4fea-b29a-3956a82e7a8b" containerName="istio-proxy"
Oct 30 20:23:51.100695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.100671    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.105918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.105891    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:51.106052 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.106000    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:51.106124 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.106092    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:51.106178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.106135    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:23:51.108970 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.108947    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"]
Oct 30 20:23:51.205439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.205398    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.205655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.205539    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.205655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.205596    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.306944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.306906    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.307143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.306967    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.307143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.307024    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.307651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.307629    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.309238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.309215    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.319686 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.319661    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn\") pod \"maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.421383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.421346    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:51.701086 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:51.701054    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"]
Oct 30 20:23:51.701346 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:23:51.701313    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac8f7f4_c5b3_4b39_a410_25bd5d7d856c.slice/crio-c2fd580bde83af0bcd83ffd660a247bdc6af4e3711cbefde1e491d07cc2015c3 WatchSource:0}: Error finding container c2fd580bde83af0bcd83ffd660a247bdc6af4e3711cbefde1e491d07cc2015c3: Status 404 returned error can't find the container with id c2fd580bde83af0bcd83ffd660a247bdc6af4e3711cbefde1e491d07cc2015c3
Oct 30 20:23:52.301736 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:52.301648    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl" event={"ID":"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c","Type":"ContainerStarted","Data":"70ac6e82366780f6d1e469c6fb7a73b53ef9888269d96cd971d603e375789ca9"}
Oct 30 20:23:52.301736 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:52.301708    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl" event={"ID":"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c","Type":"ContainerStarted","Data":"c2fd580bde83af0bcd83ffd660a247bdc6af4e3711cbefde1e491d07cc2015c3"}
Oct 30 20:23:52.323095 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:52.323048    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl" podStartSLOduration=1.323033784 podStartE2EDuration="1.323033784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:52.320447409 +0000 UTC m=+1165.223867451" watchObservedRunningTime="2026-10-30 20:23:52.323033784 +0000 UTC m=+1165.226453829"
Oct 30 20:23:53.310215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:53.310173    6058 generic.go:356] "Generic (PLEG): container finished" podID="6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" containerID="70ac6e82366780f6d1e469c6fb7a73b53ef9888269d96cd971d603e375789ca9" exitCode=0
Oct 30 20:23:53.310628 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:53.310217    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl" event={"ID":"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c","Type":"ContainerDied","Data":"70ac6e82366780f6d1e469c6fb7a73b53ef9888269d96cd971d603e375789ca9"}
Oct 30 20:23:54.467903 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.467874    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:54.637922 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.637896    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca\") pod \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") "
Oct 30 20:23:54.638082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.637938    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca\") pod \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") "
Oct 30 20:23:54.638082 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.637960    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn\") pod \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\" (UID: \"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c\") "
Oct 30 20:23:54.638382 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.638360    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca" pod "6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" (UID: "6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:54.640038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.639996    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca" pod "6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" (UID: "6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:54.640131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.639996    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn" pod "6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" (UID: "6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c"). InnerVolumeSpecName "kube-api-access-2rcjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:54.739546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.739517    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.739546 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.739546    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:54.739688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:54.739562    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c-kube-api-access-2rcjn\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:23:55.325022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:55.324987    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"
Oct 30 20:23:55.325177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:55.325042    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl" event={"ID":"6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c","Type":"ContainerDied","Data":"c2fd580bde83af0bcd83ffd660a247bdc6af4e3711cbefde1e491d07cc2015c3"}
Oct 30 20:23:55.325177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:23:55.325090    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2fd580bde83af0bcd83ffd660a247bdc6af4e3711cbefde1e491d07cc2015c3"
Oct 30 20:24:02.174106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.174075    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"]
Oct 30 20:24:02.174593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.174507    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" containerName="revoke-keys"
Oct 30 20:24:02.174593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.174518    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" containerName="revoke-keys"
Oct 30 20:24:02.174593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.174583    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" containerName="revoke-keys"
Oct 30 20:24:02.178113 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.178094    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.188274 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.188244    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:02.189164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.188997    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:24:02.189164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.189049    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:24:02.189164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.189134    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:24:02.190457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.190434    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"]
Oct 30 20:24:02.209193 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.209159    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5gc\" (UniqueName: \"kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.209310 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.209205    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.209358 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.209330    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.309963 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.309927    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5gc\" (UniqueName: \"kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.309963 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.309968    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.310157 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.309995    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.310661 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.310640    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.312245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.312225    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.322942 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.322920    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5gc\" (UniqueName: \"kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc\") pod \"maas-api-revoke-sub-96f5e1a0-qjfbb\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.493328 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.493268    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:02.770963 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:02.770935    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"]
Oct 30 20:24:02.772114 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:24:02.772088    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a455db0_ee1f_4055_9045_e8bc2abcd916.slice/crio-637168f3ca5826e06c75a4fb640a179d349829e1f74ffa513d6528f31a37d1f8 WatchSource:0}: Error finding container 637168f3ca5826e06c75a4fb640a179d349829e1f74ffa513d6528f31a37d1f8: Status 404 returned error can't find the container with id 637168f3ca5826e06c75a4fb640a179d349829e1f74ffa513d6528f31a37d1f8
Oct 30 20:24:03.379748 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:03.379711    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb" event={"ID":"5a455db0-ee1f-4055-9045-e8bc2abcd916","Type":"ContainerStarted","Data":"567423996aaea005ae2bfbd4669c1a3ea02de4a126d56ce1d3934b6162455797"}
Oct 30 20:24:03.380127 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:03.379779    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb" event={"ID":"5a455db0-ee1f-4055-9045-e8bc2abcd916","Type":"ContainerStarted","Data":"637168f3ca5826e06c75a4fb640a179d349829e1f74ffa513d6528f31a37d1f8"}
Oct 30 20:24:03.399749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:03.399706    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb" podStartSLOduration=1.3996949459999999 podStartE2EDuration="1.399694946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:03.398488668 +0000 UTC m=+1176.301908695" watchObservedRunningTime="2026-10-30 20:24:03.399694946 +0000 UTC m=+1176.303114972"
Oct 30 20:24:04.390202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:04.390161    6058 generic.go:356] "Generic (PLEG): container finished" podID="5a455db0-ee1f-4055-9045-e8bc2abcd916" containerID="567423996aaea005ae2bfbd4669c1a3ea02de4a126d56ce1d3934b6162455797" exitCode=0
Oct 30 20:24:04.390616 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:04.390248    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb" event={"ID":"5a455db0-ee1f-4055-9045-e8bc2abcd916","Type":"ContainerDied","Data":"567423996aaea005ae2bfbd4669c1a3ea02de4a126d56ce1d3934b6162455797"}
Oct 30 20:24:05.525703 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.525676    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:05.538788 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.538766    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca\") pod \"5a455db0-ee1f-4055-9045-e8bc2abcd916\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") "
Oct 30 20:24:05.538896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.538800    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca\") pod \"5a455db0-ee1f-4055-9045-e8bc2abcd916\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") "
Oct 30 20:24:05.538946 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.538901    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc\" (UniqueName: \"kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc\") pod \"5a455db0-ee1f-4055-9045-e8bc2abcd916\" (UID: \"5a455db0-ee1f-4055-9045-e8bc2abcd916\") "
Oct 30 20:24:05.539262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.539236    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca" pod "5a455db0-ee1f-4055-9045-e8bc2abcd916" (UID: "5a455db0-ee1f-4055-9045-e8bc2abcd916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:05.540788 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.540771    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc" pod "5a455db0-ee1f-4055-9045-e8bc2abcd916" (UID: "5a455db0-ee1f-4055-9045-e8bc2abcd916"). InnerVolumeSpecName "kube-api-access-hj5gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:05.540868 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.540794    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca" pod "5a455db0-ee1f-4055-9045-e8bc2abcd916" (UID: "5a455db0-ee1f-4055-9045-e8bc2abcd916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:05.639896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.639864    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj5gc\" (UniqueName: \"kubernetes.io/projected/5a455db0-ee1f-4055-9045-e8bc2abcd916-kube-api-access-hj5gc\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:05.639896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.639890    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:05.639896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:05.639900    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a455db0-ee1f-4055-9045-e8bc2abcd916-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.404822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:06.404788    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"
Oct 30 20:24:06.404982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:06.404843    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb" event={"ID":"5a455db0-ee1f-4055-9045-e8bc2abcd916","Type":"ContainerDied","Data":"637168f3ca5826e06c75a4fb640a179d349829e1f74ffa513d6528f31a37d1f8"}
Oct 30 20:24:06.404982 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:06.404891    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637168f3ca5826e06c75a4fb640a179d349829e1f74ffa513d6528f31a37d1f8"
Oct 30 20:24:07.026148 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:07.026110    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"]
Oct 30 20:24:07.030840 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:07.030806    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b937f7-gmk46"]
Oct 30 20:24:07.212734 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:07.212702    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0f2407-8e02-464f-84cc-72e94fc1244e" path="/var/lib/kubelet/pods/5b0f2407-8e02-464f-84cc-72e94fc1244e/volumes"
Oct 30 20:24:12.026617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:12.026531    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"]
Oct 30 20:24:12.030646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:12.030620    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a626424d-jbt29"]
Oct 30 20:24:13.212718 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:13.212687    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7cf03a-b513-44f7-b335-b80725980d6a" path="/var/lib/kubelet/pods/1f7cf03a-b513-44f7-b335-b80725980d6a/volumes"
Oct 30 20:24:14.326949 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:14.326919    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:24:17.491911 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:17.491872    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"]
Oct 30 20:24:17.496894 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:17.496862    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-124fd1-qbvbl"]
Oct 30 20:24:19.212471 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:19.212436    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c" path="/var/lib/kubelet/pods/6ac8f7f4-c5b3-4b39-a410-25bd5d7d856c/volumes"
Oct 30 20:24:19.522292 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:19.522205    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:24:19.655598 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:24:19.655554    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:24:24.836065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:24.836026    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"]
Oct 30 20:24:24.836584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:24.836273    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy" containerID="cri-o://fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf" gracePeriod=30
Oct 30 20:24:27.025097 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.025060    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"]
Oct 30 20:24:27.028516 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.028493    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ae2c76-mw8tk"]
Oct 30 20:24:27.211512 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.211469    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff95c6b9-d09b-45da-9f14-c00615e1ce40" path="/var/lib/kubelet/pods/ff95c6b9-d09b-45da-9f14-c00615e1ce40/volumes"
Oct 30 20:24:27.896112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.896077    6058 scope.go:122] "RemoveContainer" containerID="d56b8886447abc9217eb497cd78afebddf3d91e170f6b6536c1b87ea83c8e03e"
Oct 30 20:24:27.912521 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.912495    6058 scope.go:122] "RemoveContainer" containerID="a67a3bb56caf5b32bb161a2940570caf572ea7f8e81a4df623cdad0877d2e9ae"
Oct 30 20:24:27.921116 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.921099    6058 scope.go:122] "RemoveContainer" containerID="8ceb035aa3ff5bb20203493e46a2194a01c305efe057062875ab604e099c6853"
Oct 30 20:24:27.929296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.929280    6058 scope.go:122] "RemoveContainer" containerID="4cdce80d94d7b2141f71f36c158e9089fd6c97e09a76f0750d1961229647a853"
Oct 30 20:24:27.937057 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.937039    6058 scope.go:122] "RemoveContainer" containerID="a790555e46dd090a1fc85ef836cb261288f3881284a6c4b1416c5ad23bb68765"
Oct 30 20:24:27.946426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.946405    6058 scope.go:122] "RemoveContainer" containerID="1868719fdeb2d2ffd26722c8b7df5541264b0cbcf03d30a0ab9231b80d9cb926"
Oct 30 20:24:27.956180 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.956161    6058 scope.go:122] "RemoveContainer" containerID="2245458b4d271fbeafb6d93155269d24b31da0a567983a45b6cb07a90102cc5d"
Oct 30 20:24:27.966120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.966095    6058 scope.go:122] "RemoveContainer" containerID="f93f046fab3f7b99fdb20e224dbe3ff0b392a062d9ba64910dfcef55a6149a29"
Oct 30 20:24:27.979143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.979115    6058 scope.go:122] "RemoveContainer" containerID="688b84f88a0d565691ce979d8be3f9a947f28e319662a531266570f97f10a3a1"
Oct 30 20:24:27.990119 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:27.990090    6058 scope.go:122] "RemoveContainer" containerID="d1b662d30d6dcd3237ddfce6b97d9fe0d87d9de3ddaae07d227541c1411a3951"
Oct 30 20:24:30.347571 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.347540    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:24:30.538428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538344    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538397    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538425    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538477    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538507    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538566    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538617    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538661    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538696 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538690    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert\") pod \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\" (UID: \"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef\") "
Oct 30 20:24:30.538984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.538717    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:30.539049 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.539018    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:30.539111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.539047    6058 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.539111 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.539028    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:30.539211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.539161    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:30.539338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.539307    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:30.540700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.540671    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "kube-api-access-9cnxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:30.540843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.540823    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:24:30.540906 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.540873    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:30.540987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.540968    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token" pod "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" (UID: "bf5283e6-a3a3-43ea-bc08-08cfbadd65ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:30.628028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.627590    6058 generic.go:356] "Generic (PLEG): container finished" podID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerID="fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf" exitCode=0
Oct 30 20:24:30.628028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.627742    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" event={"ID":"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef","Type":"ContainerDied","Data":"fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf"}
Oct 30 20:24:30.628028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.627780    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb" event={"ID":"bf5283e6-a3a3-43ea-bc08-08cfbadd65ef","Type":"ContainerDied","Data":"11c436b849cf9d18426f924d8681d039ec44f1ccb1214b236deb08503506c305"}
Oct 30 20:24:30.628028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.627802    6058 scope.go:122] "RemoveContainer" containerID="fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf"
Oct 30 20:24:30.628028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.627928    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"
Oct 30 20:24:30.639726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.639705    6058 scope.go:122] "RemoveContainer" containerID="fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf"
Oct 30 20:24:30.640029 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:24:30.639991    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf\": container with ID starting with fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf not found: ID does not exist" containerID="fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf"
Oct 30 20:24:30.640088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640039    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf"} err="failed to get container status \"fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf\": rpc error: code = NotFound desc = could not find container \"fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf\": container with ID starting with fbd05520299b43bd53f8e4ae72651d3b6f45fe3d92793fa71a004d9e6e58cedf not found: ID does not exist"
Oct 30 20:24:30.640134 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640117    6058 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-data\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640135    6058 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-workload-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640144    6058 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-credential-socket\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640153    6058 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istiod-ca-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640161    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640291 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640170    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cnxc\" (UniqueName: \"kubernetes.io/projected/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-kube-api-access-9cnxc\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640291 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640208    6058 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-envoy\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.640291 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.640222    6058 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef-istio-podinfo\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:30.688030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.687060    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"]
Oct 30 20:24:30.715027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:30.691117    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-124fd1-openshift-default-5b554bcd59-nwpcb"]
Oct 30 20:24:31.211826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:31.211787    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" path="/var/lib/kubelet/pods/bf5283e6-a3a3-43ea-bc08-08cfbadd65ef/volumes"
Oct 30 20:24:34.026121 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:34.026083    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"]
Oct 30 20:24:34.029893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:34.029870    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba0d32f-pptlw"]
Oct 30 20:24:35.211323 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:35.211292    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630949d5-5f6e-4227-9aaa-5db2731dff90" path="/var/lib/kubelet/pods/630949d5-5f6e-4227-9aaa-5db2731dff90/volumes"
Oct 30 20:24:45.036670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:45.036638    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"]
Oct 30 20:24:45.046249 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:45.046220    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe02ee94-plfhl"]
Oct 30 20:24:45.211257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:45.211225    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3" path="/var/lib/kubelet/pods/0d62e78a-2a6b-4a62-a50c-d4bc4fced3f3/volumes"
Oct 30 20:24:49.658630 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:24:49.658588    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:24:58.024674 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:58.024634    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"]
Oct 30 20:24:58.028391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:58.028361    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76a8a92-m46bv"]
Oct 30 20:24:59.212065 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:24:59.212030    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f166e04-09ad-4671-97b5-9c3ba29bb2d3" path="/var/lib/kubelet/pods/7f166e04-09ad-4671-97b5-9c3ba29bb2d3/volumes"
Oct 30 20:25:03.032200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:03.032159    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"]
Oct 30 20:25:03.036481 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:03.036455    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b086870-bb7sb"]
Oct 30 20:25:03.211548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:03.211517    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed361e74-3609-4e97-b768-f7d4d83c396c" path="/var/lib/kubelet/pods/ed361e74-3609-4e97-b768-f7d4d83c396c/volumes"
Oct 30 20:25:10.025501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:10.025470    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"]
Oct 30 20:25:10.030896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:10.030867    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00fd4b3-55zpx"]
Oct 30 20:25:11.211601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:11.211569    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1115fe-1333-41d7-b0ed-4acfa8789e80" path="/var/lib/kubelet/pods/9c1115fe-1333-41d7-b0ed-4acfa8789e80/volumes"
Oct 30 20:25:19.655315 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:25:19.655279    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:25:22.027530 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:22.027497    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"]
Oct 30 20:25:22.031325 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:22.031296    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b85c442-rtzhj"]
Oct 30 20:25:23.035150 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:23.035110    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"]
Oct 30 20:25:23.038548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:23.038522    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14f5b7-nmwnf"]
Oct 30 20:25:23.217926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:23.217892    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5" path="/var/lib/kubelet/pods/9ffe7b56-1cd5-4a18-84b8-d0f37b0173d5/volumes"
Oct 30 20:25:23.218367 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:23.218350    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea350170-1c31-4533-bc57-f3ba1a819c2b" path="/var/lib/kubelet/pods/ea350170-1c31-4533-bc57-f3ba1a819c2b/volumes"
Oct 30 20:25:28.083709 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:28.083673    6058 scope.go:122] "RemoveContainer" containerID="26d55724660d1fb9c82d51e46cafdfab9e49abf0c3f3ec900d2ab5fbf8cd3ec5"
Oct 30 20:25:28.092839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:28.092819    6058 scope.go:122] "RemoveContainer" containerID="1dd0b2bea4d82a537dfd84106efeeb45f238062ba387152d607b1e4aa81c0a19"
Oct 30 20:25:28.101172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:28.101152    6058 scope.go:122] "RemoveContainer" containerID="6d386e0eef22d8304b90848c172a5a07d4cb88d55bfc1390164dacebf5f0364c"
Oct 30 20:25:28.109385 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:28.109363    6058 scope.go:122] "RemoveContainer" containerID="0a24c0cf959b18c99fb044083432d5ffa8b8811736058cd7856a8b92b6f5a8ac"
Oct 30 20:25:30.945983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:30.945954    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:25:41.026563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:41.026491    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"]
Oct 30 20:25:41.031784 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:41.031758    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b2d9e36-gm4rn"]
Oct 30 20:25:41.212936 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:41.212901    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b00dab7-7025-4f89-9f4a-97c6a41a7bbb" path="/var/lib/kubelet/pods/1b00dab7-7025-4f89-9f4a-97c6a41a7bbb/volumes"
Oct 30 20:25:43.025330 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:43.025292    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"]
Oct 30 20:25:43.028437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:43.028412    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0636f71-9tvl9"]
Oct 30 20:25:43.210922 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:43.210885    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b52bde-0263-42b9-b4de-81ee45550d08" path="/var/lib/kubelet/pods/79b52bde-0263-42b9-b4de-81ee45550d08/volumes"
Oct 30 20:25:46.354735 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:46.354691    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:25:49.746550 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:25:49.746510    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:25:49.927386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:49.927342    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:25:51.026964 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:51.026928    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"]
Oct 30 20:25:51.030248 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:51.030224    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9801bdc1-mrhql"]
Oct 30 20:25:51.213845 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:51.213805    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9013f712-b991-47b6-8100-615475806ee8" path="/var/lib/kubelet/pods/9013f712-b991-47b6-8100-615475806ee8/volumes"
Oct 30 20:25:55.025451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:55.025414    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"]
Oct 30 20:25:55.028930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:55.028902    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769d6ada-ltvrq"]
Oct 30 20:25:55.210761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:25:55.210732    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179272ea-cfa3-474e-bf19-b34cef4a9bc1" path="/var/lib/kubelet/pods/179272ea-cfa3-474e-bf19-b34cef4a9bc1/volumes"
Oct 30 20:26:05.259310 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:05.259267    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:26:11.035471 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:11.035426    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"]
Oct 30 20:26:11.039613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:11.039586    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409e39e9-4kcqf"]
Oct 30 20:26:11.211859 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:11.211823    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04c7537-7eea-41b6-965e-efeaa585c8aa" path="/var/lib/kubelet/pods/a04c7537-7eea-41b6-965e-efeaa585c8aa/volumes"
Oct 30 20:26:18.020636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:18.020598    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:26:19.736245 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:26:19.736203    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:26:23.030724 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:23.030688    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"]
Oct 30 20:26:23.033711 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:23.033686    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365edb9-ck2gg"]
Oct 30 20:26:23.211839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:23.211809    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b53f23-768a-464c-8486-a2e38b90629d" path="/var/lib/kubelet/pods/c5b53f23-768a-464c-8486-a2e38b90629d/volumes"
Oct 30 20:26:25.024499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:25.024468    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"]
Oct 30 20:26:25.027994 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:25.027969    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc1b223-thcrg"]
Oct 30 20:26:25.211412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:25.211384    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c" path="/var/lib/kubelet/pods/adf05cc8-c8e7-4a08-b7f1-c1ef5983f07c/volumes"
Oct 30 20:26:27.026135 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:27.026097    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"]
Oct 30 20:26:27.029827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:27.029799    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582d838-vfn24"]
Oct 30 20:26:27.211855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:27.211818    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9857d9c1-7d1d-45e1-8340-076f34f189c3" path="/var/lib/kubelet/pods/9857d9c1-7d1d-45e1-8340-076f34f189c3/volumes"
Oct 30 20:26:28.149584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.149552    6058 scope.go:122] "RemoveContainer" containerID="721da78f755802c7d991d4e7538420cfba909f0f5650c4d073d40938572a5c6c"
Oct 30 20:26:28.158540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.158521    6058 scope.go:122] "RemoveContainer" containerID="af1dd9b16900368ab87876316ab955e12f163ac15d567bacb8f8d9554b2790e5"
Oct 30 20:26:28.169919 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.169900    6058 scope.go:122] "RemoveContainer" containerID="43c6a495190b8d8256a6588bc8a9f8e0ed3c8ddf0f9eec9fd52fdc1d96286ba7"
Oct 30 20:26:28.178262 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.178241    6058 scope.go:122] "RemoveContainer" containerID="8cbbeed94598de35f3fa7db071a62a074d8e4b99c0402ec165a1a9a5cc482f65"
Oct 30 20:26:28.186276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.186252    6058 scope.go:122] "RemoveContainer" containerID="e8a44507f961164c433b7fced6a58dbe3731438cf33f72fba8deabc3817403e0"
Oct 30 20:26:28.193742 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.193710    6058 scope.go:122] "RemoveContainer" containerID="fd1631d011fcd1b11931f0b4bf6324955b69f920e4fadb97e139ad1f19a9dfc6"
Oct 30 20:26:28.201295 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:28.201280    6058 scope.go:122] "RemoveContainer" containerID="aaca24a1b2f4f59ad808f372ec11e7fd9ff0a86253430a9d9bc474dbcaa323d4"
Oct 30 20:26:31.036859 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:31.036826    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"]
Oct 30 20:26:31.042056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:31.042025    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ac179e-dfw6d"]
Oct 30 20:26:31.219484 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:31.219449    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8f7190-e22e-4e8d-b1c5-13b1b6e87762" path="/var/lib/kubelet/pods/9d8f7190-e22e-4e8d-b1c5-13b1b6e87762/volumes"
Oct 30 20:26:38.025141 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:38.025105    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"]
Oct 30 20:26:38.030497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:38.030471    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc831e2-stpld"]
Oct 30 20:26:39.155444 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:39.155407    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:26:39.211846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:39.211779    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f9abdb-d115-4fb5-aa0c-bdeac194a503" path="/var/lib/kubelet/pods/a9f9abdb-d115-4fb5-aa0c-bdeac194a503/volumes"
Oct 30 20:26:41.037032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:41.028924    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"]
Oct 30 20:26:41.041304 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:41.041241    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4929e599-67wch"]
Oct 30 20:26:41.239650 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:41.239601    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289ded7b-1109-4a03-8b4a-703ad84ce11c" path="/var/lib/kubelet/pods/289ded7b-1109-4a03-8b4a-703ad84ce11c/volumes"
Oct 30 20:26:43.027995 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:43.027962    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"]
Oct 30 20:26:43.037924 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:43.037889    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7af2aa-pnmxv"]
Oct 30 20:26:43.215610 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:43.215580    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e971d5d5-5122-42d1-bf0b-62eea12b4196" path="/var/lib/kubelet/pods/e971d5d5-5122-42d1-bf0b-62eea12b4196/volumes"
Oct 30 20:26:47.025015 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:47.024981    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"]
Oct 30 20:26:47.027603 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:47.027573    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e0db5c-7jsdj"]
Oct 30 20:26:47.212114 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:47.212081    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165323da-d482-48d7-8616-0f20cef73c1f" path="/var/lib/kubelet/pods/165323da-d482-48d7-8616-0f20cef73c1f/volumes"
Oct 30 20:26:49.556660 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:49.556615    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:26:49.656064 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:26:49.656001    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:26:54.181040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:54.168690    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:26:55.024943 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:55.024905    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"]
Oct 30 20:26:55.028574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:55.028546    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c14be32-zvkgs"]
Oct 30 20:26:55.211897 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:26:55.211864    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c228f1-61b6-453c-bfe9-8e642629445a" path="/var/lib/kubelet/pods/a0c228f1-61b6-453c-bfe9-8e642629445a/volumes"
Oct 30 20:27:00.026231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:00.026197    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"]
Oct 30 20:27:00.030292 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:00.030269    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86442197-tk6jt"]
Oct 30 20:27:01.211821 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:01.211786    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014962b9-b1cd-4f86-b667-bdd3f5ca24e6" path="/var/lib/kubelet/pods/014962b9-b1cd-4f86-b667-bdd3f5ca24e6/volumes"
Oct 30 20:27:10.338539 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:10.338455    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:27:13.729032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:13.728947    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:27:14.959031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:14.958107    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ls9cm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:27:14.959031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:14.958171    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ls9cm" podUID="dc5e977f-8820-4bd8-a733-4c35785d3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:27:19.654959 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:27:19.654918    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:27:21.036831 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.036797    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"]
Oct 30 20:27:21.044673 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.044650    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"]
Oct 30 20:27:21.048107 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.048084    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"]
Oct 30 20:27:21.049861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.049828    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b522314-n2fg2"]
Oct 30 20:27:21.051746 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.051721    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89c471f-6rhpc"]
Oct 30 20:27:21.053252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.053229    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7fc1f8-l6xhd"]
Oct 30 20:27:21.211690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.211658    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b29394f-045e-495b-ab74-adaadec9f485" path="/var/lib/kubelet/pods/4b29394f-045e-495b-ab74-adaadec9f485/volumes"
Oct 30 20:27:21.212096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.212082    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e0637e-89e4-4f94-90e0-12524247f76c" path="/var/lib/kubelet/pods/62e0637e-89e4-4f94-90e0-12524247f76c/volumes"
Oct 30 20:27:21.212454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:21.212441    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42fc25c-a8fe-402e-841b-abd0014266a2" path="/var/lib/kubelet/pods/a42fc25c-a8fe-402e-841b-abd0014266a2/volumes"
Oct 30 20:27:28.247524 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.247493    6058 scope.go:122] "RemoveContainer" containerID="596b9223f19566ed59f13612713e9a12e75dd6fdcee9de89393de7daac49e371"
Oct 30 20:27:28.260154 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.260124    6058 scope.go:122] "RemoveContainer" containerID="2be5ade4ccf13b13b804e2c2397201a872b3b000fe73c70fb95a6c046ea330b6"
Oct 30 20:27:28.269190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.269169    6058 scope.go:122] "RemoveContainer" containerID="a94c3c083febd1a0cc50ecc3950fc89c504b84ffa848db9721ee0578c410edfc"
Oct 30 20:27:28.276772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.276754    6058 scope.go:122] "RemoveContainer" containerID="1de8f8561bd2a6b66e0f87c7091385972bdb454414979b6f777eec0bdc52414a"
Oct 30 20:27:28.284679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.284649    6058 scope.go:122] "RemoveContainer" containerID="36331a4c857fe962f3f518d9f0c4140aa0d4584a4f2f028521a7ac023bc90064"
Oct 30 20:27:28.292344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.292328    6058 scope.go:122] "RemoveContainer" containerID="d5bdac4892fd54fdcc90e352c0626d64537f0c21699a0449b796dfac612a9883"
Oct 30 20:27:28.300468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.300453    6058 scope.go:122] "RemoveContainer" containerID="4b168548d8178cac89c57eb229cc025612e7cd6b50ca7f1e7b1ce7c6d620d76c"
Oct 30 20:27:28.307884 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.307868    6058 scope.go:122] "RemoveContainer" containerID="3362363d0717ffa337ed5bbff4c5d8e9b112aec51125c9b48cbb00d8307fbad9"
Oct 30 20:27:28.316537 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.316519    6058 scope.go:122] "RemoveContainer" containerID="65bf1e498a1fbcbcfa7f456728fa73989b3d0291b442695b02108123752cee8a"
Oct 30 20:27:28.679803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:28.679770    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:27:29.026599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:29.026518    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"]
Oct 30 20:27:29.030570 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:29.030544    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"]
Oct 30 20:27:29.032207 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:29.032138    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee67d537-pkmrh"]
Oct 30 20:27:29.033694 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:29.033669    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc82938-r5ntp"]
Oct 30 20:27:29.213231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:29.213198    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a3c647-5851-4cd4-a366-a154129abc8e" path="/var/lib/kubelet/pods/71a3c647-5851-4cd4-a366-a154129abc8e/volumes"
Oct 30 20:27:29.213604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:29.213590    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49d1f64-f2f4-4d21-938b-16fd7781b8ec" path="/var/lib/kubelet/pods/a49d1f64-f2f4-4d21-938b-16fd7781b8ec/volumes"
Oct 30 20:27:32.934611 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:32.934578    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:27:37.025953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.025914    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"]
Oct 30 20:27:37.031483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.031455    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"]
Oct 30 20:27:37.034968 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.034944    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"]
Oct 30 20:27:37.036128 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.036110    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d19f1e0-pvwdc"]
Oct 30 20:27:37.037182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.037162    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cadc29-kr44l"]
Oct 30 20:27:37.039701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.039681    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f62ea98-27szt"]
Oct 30 20:27:37.211535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.211509    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c261b9-b134-4528-b9de-3e64428aa26c" path="/var/lib/kubelet/pods/14c261b9-b134-4528-b9de-3e64428aa26c/volumes"
Oct 30 20:27:37.211909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.211896    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb516c6-e0cd-4b82-9a7c-8a13b9d74690" path="/var/lib/kubelet/pods/4eb516c6-e0cd-4b82-9a7c-8a13b9d74690/volumes"
Oct 30 20:27:37.212275 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:37.212262    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b29b091-7e73-4680-8df9-64dfca7f537d" path="/var/lib/kubelet/pods/9b29b091-7e73-4680-8df9-64dfca7f537d/volumes"
Oct 30 20:27:46.028064 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:46.028033    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"]
Oct 30 20:27:46.033973 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:46.033945    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fd8f84-mqpdl"]
Oct 30 20:27:46.840270 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:46.840205    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:27:47.211178 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:47.211142    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676c126a-f54a-491d-a9e7-def447e917b0" path="/var/lib/kubelet/pods/676c126a-f54a-491d-a9e7-def447e917b0/volumes"
Oct 30 20:27:49.665160 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:27:49.665102    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:27:50.228900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:27:50.228856    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:00.030910 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:00.030873    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"]
Oct 30 20:28:00.033806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:00.033781    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d903d04e-v8psz"]
Oct 30 20:28:01.210893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:01.210855    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60a265c-8d1e-4a2b-8414-aefe15661c18" path="/var/lib/kubelet/pods/b60a265c-8d1e-4a2b-8414-aefe15661c18/volumes"
Oct 30 20:28:04.867139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:04.867105    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:13.473937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.473900    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"]
Oct 30 20:28:13.474435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.474123    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv" podUID="6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" containerName="discovery" containerID="cri-o://8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912" gracePeriod=30
Oct 30 20:28:13.729480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.729425    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:28:13.838211 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838180    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.838383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838226    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.838383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838268    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.838383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838321    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.838383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838349    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.838383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838375    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.838675 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.838434    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig\") pod \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\" (UID: \"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c\") "
Oct 30 20:28:13.839204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.839164    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:13.840515 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.840490    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:13.840652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.840627    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:13.841016 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.840976    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:13.841083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.840996    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:13.841083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.840996    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:13.841083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.840986    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx" pod "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" (UID: "6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c"). InnerVolumeSpecName "kube-api-access-7rphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:13.940191 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940156    6058 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-kubeconfig\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:13.940191 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940184    6058 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-dns-cert\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:13.940191 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940194    6058 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-cacerts\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:13.940421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940203    6058 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-local-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:13.940421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940212    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rphx\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-kube-api-access-7rphx\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:13.940421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940222    6058 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-token\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:13.940421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:13.940232    6058 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c-istio-csr-ca-configmap\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:14.166473 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.166440    6058 generic.go:356] "Generic (PLEG): container finished" podID="6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" containerID="8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912" exitCode=0
Oct 30 20:28:14.166651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.166493    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"
Oct 30 20:28:14.166651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.166493    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv" event={"ID":"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c","Type":"ContainerDied","Data":"8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912"}
Oct 30 20:28:14.166651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.166620    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv" event={"ID":"6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c","Type":"ContainerDied","Data":"957e256d1efdf02e95ffeccfe8ef72bf97709aef2d7c679dc08406e5688d0186"}
Oct 30 20:28:14.166651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.166650    6058 scope.go:122] "RemoveContainer" containerID="8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912"
Oct 30 20:28:14.176186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.176169    6058 scope.go:122] "RemoveContainer" containerID="8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912"
Oct 30 20:28:14.176439 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:28:14.176423    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912\": container with ID starting with 8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912 not found: ID does not exist" containerID="8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912"
Oct 30 20:28:14.176506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.176452    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912"} err="failed to get container status \"8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912\": rpc error: code = NotFound desc = could not find container \"8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912\": container with ID starting with 8ea97e0de4a2c96c11b9beb42ca871469baff2d753e9171b280c7149163bf912 not found: ID does not exist"
Oct 30 20:28:14.191458 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.191434    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"]
Oct 30 20:28:14.195812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:14.195793    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hm6jv"]
Oct 30 20:28:15.212148 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:15.212111    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" path="/var/lib/kubelet/pods/6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c/volumes"
Oct 30 20:28:15.571435 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:15.571344    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:19.634023 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:19.633976    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:19.655441 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:28:19.655401    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:28:22.912349 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912314    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"]
Oct 30 20:28:22.912721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912707    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5a455db0-ee1f-4055-9045-e8bc2abcd916" containerName="revoke-keys"
Oct 30 20:28:22.912721 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912718    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a455db0-ee1f-4055-9045-e8bc2abcd916" containerName="revoke-keys"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912729    6058 cpu_manager.go:414] "containerMap: removing container" podUID="6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" containerName="discovery"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912735    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" containerName="discovery"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912746    6058 cpu_manager.go:414] "containerMap: removing container" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912752    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912809    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5283e6-a3a3-43ea-bc08-08cfbadd65ef" containerName="istio-proxy"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912822    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a455db0-ee1f-4055-9045-e8bc2abcd916" containerName="revoke-keys"
Oct 30 20:28:22.912827 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.912828    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="6955dd8c-7e85-4bbb-a1f5-2a59aeae8b5c" containerName="discovery"
Oct 30 20:28:22.916230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.916214    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:22.920700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.920679    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:22.921528 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.921511    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:22.921603 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.921587    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:22.921684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.921669    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:28:22.925553 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:22.925529    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"]
Oct 30 20:28:23.023083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.023053    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"]
Oct 30 20:28:23.028366 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.028341    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e5915f-tbc2f"]
Oct 30 20:28:23.064611 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.064575    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.064730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.064642    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.064779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.064747    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.166029 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.165952    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.166151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.166046    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.166151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.166083    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.166664 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.166642    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.168343 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.168325    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.178951 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.178931    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv\") pod \"maas-api-revoke-sub-3d4156f1-cmtvc\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.212391 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.212368    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4e032a-d5c4-418c-97a2-55518640ada6" path="/var/lib/kubelet/pods/2e4e032a-d5c4-418c-97a2-55518640ada6/volumes"
Oct 30 20:28:23.234474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.234451    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:23.508361 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:23.508335    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"]
Oct 30 20:28:23.510998 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:28:23.510973    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726b166b_f447_406b_8e50_38dc99f056a9.slice/crio-74c87f64acb8e91a851622069ce9ade7fdc369654ba971ad169dcd74e2f1b479 WatchSource:0}: Error finding container 74c87f64acb8e91a851622069ce9ade7fdc369654ba971ad169dcd74e2f1b479: Status 404 returned error can't find the container with id 74c87f64acb8e91a851622069ce9ade7fdc369654ba971ad169dcd74e2f1b479
Oct 30 20:28:24.251457 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:24.251421    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc" event={"ID":"726b166b-f447-406b-8e50-38dc99f056a9","Type":"ContainerStarted","Data":"55532976c2c69f36c6370cdfcc8f900897b86e4516651b17cb36e36539927a50"}
Oct 30 20:28:24.251818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:24.251496    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc" event={"ID":"726b166b-f447-406b-8e50-38dc99f056a9","Type":"ContainerStarted","Data":"74c87f64acb8e91a851622069ce9ade7fdc369654ba971ad169dcd74e2f1b479"}
Oct 30 20:28:24.271945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:24.271893    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc" podStartSLOduration=2.271880586 podStartE2EDuration="2.271880586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:24.270231985 +0000 UTC m=+1437.173652013" watchObservedRunningTime="2026-10-30 20:28:24.271880586 +0000 UTC m=+1437.175300612"
Oct 30 20:28:25.257497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:25.257462    6058 generic.go:356] "Generic (PLEG): container finished" podID="726b166b-f447-406b-8e50-38dc99f056a9" containerID="55532976c2c69f36c6370cdfcc8f900897b86e4516651b17cb36e36539927a50" exitCode=0
Oct 30 20:28:25.257865 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:25.257543    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc" event={"ID":"726b166b-f447-406b-8e50-38dc99f056a9","Type":"ContainerDied","Data":"55532976c2c69f36c6370cdfcc8f900897b86e4516651b17cb36e36539927a50"}
Oct 30 20:28:26.393587 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.393557    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:26.495737 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.495712    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv\") pod \"726b166b-f447-406b-8e50-38dc99f056a9\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") "
Oct 30 20:28:26.495900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.495764    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca\") pod \"726b166b-f447-406b-8e50-38dc99f056a9\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") "
Oct 30 20:28:26.495900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.495797    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca\") pod \"726b166b-f447-406b-8e50-38dc99f056a9\" (UID: \"726b166b-f447-406b-8e50-38dc99f056a9\") "
Oct 30 20:28:26.496202 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.496179    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca" pod "726b166b-f447-406b-8e50-38dc99f056a9" (UID: "726b166b-f447-406b-8e50-38dc99f056a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:26.497738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.497706    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca" pod "726b166b-f447-406b-8e50-38dc99f056a9" (UID: "726b166b-f447-406b-8e50-38dc99f056a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:26.497835 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.497749    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv" pod "726b166b-f447-406b-8e50-38dc99f056a9" (UID: "726b166b-f447-406b-8e50-38dc99f056a9"). InnerVolumeSpecName "kube-api-access-965sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:26.596749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.596662    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:26.596749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.596696    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726b166b-f447-406b-8e50-38dc99f056a9-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:26.596749 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:26.596712    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/726b166b-f447-406b-8e50-38dc99f056a9-kube-api-access-965sv\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:27.269703 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:27.269679    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"
Oct 30 20:28:27.269832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:27.269718    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc" event={"ID":"726b166b-f447-406b-8e50-38dc99f056a9","Type":"ContainerDied","Data":"74c87f64acb8e91a851622069ce9ade7fdc369654ba971ad169dcd74e2f1b479"}
Oct 30 20:28:27.269832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:27.269766    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c87f64acb8e91a851622069ce9ade7fdc369654ba971ad169dcd74e2f1b479"
Oct 30 20:28:28.370525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.370490    6058 scope.go:122] "RemoveContainer" containerID="cd00e8c757bc4e1727be7ce3412e9ae2b971170352bff6ad12213e741d19cad8"
Oct 30 20:28:28.378869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.378851    6058 scope.go:122] "RemoveContainer" containerID="64123ff8be9619801ed76d3eb0ef3b9d25ffd505e80bddcacbea1638615ba854"
Oct 30 20:28:28.386322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.386303    6058 scope.go:122] "RemoveContainer" containerID="f587ea1735cfd130ae2e9af6d3436aad4c9344bdba4ebd88dbc7f8306dee7b46"
Oct 30 20:28:28.393953 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.393926    6058 scope.go:122] "RemoveContainer" containerID="b5a761b894cfd362a5075430b935799d494c23d78587faa89aa8ddd2b9f88da3"
Oct 30 20:28:28.401257 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.401240    6058 scope.go:122] "RemoveContainer" containerID="8d37b450bfd5d26c096bdb159ca1523aa276c2758df4d866d016980833e4c165"
Oct 30 20:28:28.408430 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.408413    6058 scope.go:122] "RemoveContainer" containerID="b184411df43e9fa11c3829198a4fc71dbe2bd13219050b57982f93e1485c583f"
Oct 30 20:28:28.415680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.415663    6058 scope.go:122] "RemoveContainer" containerID="36e79f688399bf617588cff195d7852c15cf344599ae16afa6cbe7b30f23accf"
Oct 30 20:28:28.422660 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.422643    6058 scope.go:122] "RemoveContainer" containerID="b3aaa633dd670f61f5b3cee0eb8cd8423ca4da4378f77e7b219feba036fbf57c"
Oct 30 20:28:28.429474 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.429456    6058 scope.go:122] "RemoveContainer" containerID="ad0387d7c9c899f92458fd4d132c0b0a97181e83cdf36e14d35a83e4f32b6ff6"
Oct 30 20:28:28.436458 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.436443    6058 scope.go:122] "RemoveContainer" containerID="f286e0cfdbf4a92168d63453976a3c2806b9d15455e13185683ea8f9c147d3a8"
Oct 30 20:28:28.445997 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.445981    6058 scope.go:122] "RemoveContainer" containerID="ef6268aa97cfe0a7465a0d66c2a2e19a1cca36d136a52bd70db02315fd14cb8b"
Oct 30 20:28:28.453930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.453915    6058 scope.go:122] "RemoveContainer" containerID="0a7808c6c2f09f9b6fb49a2494a6b3fab63f49a8618d234a79acd5f3406707d6"
Oct 30 20:28:28.460793 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.460768    6058 scope.go:122] "RemoveContainer" containerID="bf307cceb7ee8842ca12d20ba0fb5156135ace5f632569c4edf9d70cd8fde5c6"
Oct 30 20:28:28.468052 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:28.468036    6058 scope.go:122] "RemoveContainer" containerID="19e0784c7584e6bc704b42bc75d1feb524fac7c32bef4b4495a9444862c95074"
Oct 30 20:28:34.236210 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:34.236175    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:37.840451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:37.840368    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:43.578331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.578293    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"]
Oct 30 20:28:43.587144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.587115    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmtvc"]
Oct 30 20:28:43.613296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.613268    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"]
Oct 30 20:28:43.613678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.613665    6058 cpu_manager.go:414] "containerMap: removing container" podUID="726b166b-f447-406b-8e50-38dc99f056a9" containerName="revoke-keys"
Oct 30 20:28:43.613729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.613680    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726b166b-f447-406b-8e50-38dc99f056a9" containerName="revoke-keys"
Oct 30 20:28:43.613773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.613744    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="726b166b-f447-406b-8e50-38dc99f056a9" containerName="revoke-keys"
Oct 30 20:28:43.620331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.620307    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.636266 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.636246    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:43.639918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.639903    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:43.641955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.641929    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"]
Oct 30 20:28:43.648826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.648797    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:43.648950 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.648884    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:28:43.650439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.650391    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.650574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.650526    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2z8l\" (UniqueName: \"kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.650574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.650573    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.751369 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.751337    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.751505 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.751394    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2z8l\" (UniqueName: \"kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.751505 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.751425    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.752165 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.752145    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.753666 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.753643    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.769523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.769498    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2z8l\" (UniqueName: \"kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l\") pod \"maas-api-revoke-sub-3d4156f1-2bs7p\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:43.933583 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:43.933549    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:44.211927 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:44.211873    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"]
Oct 30 20:28:44.214526 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:28:44.214501    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd3fcd4_24cd_41f2_bed9_0b5d3e7e0251.slice/crio-ee2fc8ac7764ff8f2f1c4ba78ff8de37a80de2371314e0155bce1c7f9c871021 WatchSource:0}: Error finding container ee2fc8ac7764ff8f2f1c4ba78ff8de37a80de2371314e0155bce1c7f9c871021: Status 404 returned error can't find the container with id ee2fc8ac7764ff8f2f1c4ba78ff8de37a80de2371314e0155bce1c7f9c871021
Oct 30 20:28:44.353334 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:44.353296    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p" event={"ID":"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251","Type":"ContainerStarted","Data":"ee2fc8ac7764ff8f2f1c4ba78ff8de37a80de2371314e0155bce1c7f9c871021"}
Oct 30 20:28:45.211186 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:45.211150    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726b166b-f447-406b-8e50-38dc99f056a9" path="/var/lib/kubelet/pods/726b166b-f447-406b-8e50-38dc99f056a9/volumes"
Oct 30 20:28:45.359516 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:45.359487    6058 generic.go:356] "Generic (PLEG): container finished" podID="0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" containerID="8ab5ccebd7a234bcc115fdd3b830677a18ffbe3f1d5af87d9f1e62c222f985f8" exitCode=0
Oct 30 20:28:45.359742 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:45.359564    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p" event={"ID":"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251","Type":"ContainerDied","Data":"8ab5ccebd7a234bcc115fdd3b830677a18ffbe3f1d5af87d9f1e62c222f985f8"}
Oct 30 20:28:46.492779 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.492756    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:46.575358 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.575333    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca\") pod \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") "
Oct 30 20:28:46.575502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.575364    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca\") pod \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") "
Oct 30 20:28:46.575502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.575438    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l\" (UniqueName: \"kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l\") pod \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\" (UID: \"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251\") "
Oct 30 20:28:46.575795 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.575775    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca" pod "0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" (UID: "0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:46.577356 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.577330    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca" pod "0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" (UID: "0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:46.577440 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.577381    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l" pod "0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" (UID: "0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251"). InnerVolumeSpecName "kube-api-access-r2z8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:46.676549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.676473    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2z8l\" (UniqueName: \"kubernetes.io/projected/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-kube-api-access-r2z8l\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:46.676549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.676503    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:46.676549 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:46.676513    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.370200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:47.370166    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"
Oct 30 20:28:47.370200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:47.370179    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p" event={"ID":"0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251","Type":"ContainerDied","Data":"ee2fc8ac7764ff8f2f1c4ba78ff8de37a80de2371314e0155bce1c7f9c871021"}
Oct 30 20:28:47.370394 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:47.370256    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2fc8ac7764ff8f2f1c4ba78ff8de37a80de2371314e0155bce1c7f9c871021"
Oct 30 20:28:49.657629 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:28:49.657597    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:28:54.346313 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:54.346277    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:28:57.644880 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:28:57.644843    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:29:06.028566 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.028528    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"]
Oct 30 20:29:06.032317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.032284    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f5e1a0-qjfbb"]
Oct 30 20:29:06.958843 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.958799    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"]
Oct 30 20:29:06.959247 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.959232    6058 cpu_manager.go:414] "containerMap: removing container" podUID="0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" containerName="revoke-keys"
Oct 30 20:29:06.959302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.959250    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" containerName="revoke-keys"
Oct 30 20:29:06.959339 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.959310    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" containerName="revoke-keys"
Oct 30 20:29:06.962384 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.962364    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:06.971385 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:06.971354    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"]
Oct 30 20:29:07.007228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.007200    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-dad4d6b6-dockercfg-gm6w4\""
Oct 30 20:29:07.013780 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.013755    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-402199\""
Oct 30 20:29:07.015561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.015529    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"]
Oct 30 20:29:07.019347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.019327    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.030248 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.030226    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"]
Oct 30 20:29:07.060160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.060132    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.060306 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.060200    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96d7k\" (UniqueName: \"kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k\") pod \"payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.161393 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.161343    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x468\" (UniqueName: \"kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468\") pod \"payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.161591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.161476    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.161591 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.161539    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.161680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.161604    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96d7k\" (UniqueName: \"kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k\") pod \"payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.162837 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.162807    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.174956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.174929    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96d7k\" (UniqueName: \"kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k\") pod \"payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.212878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.212815    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a455db0-ee1f-4055-9045-e8bc2abcd916" path="/var/lib/kubelet/pods/5a455db0-ee1f-4055-9045-e8bc2abcd916/volumes"
Oct 30 20:29:07.262985 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.262959    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.263096 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.263029    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x468\" (UniqueName: \"kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468\") pod \"payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.263601 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.263582    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.276236 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.276211    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:07.276689 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.276665    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x468\" (UniqueName: \"kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468\") pod \"payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.332706 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.332653    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:07.553441 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.553353    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"]
Oct 30 20:29:07.557405 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:29:07.557377    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20f3399_aa21_4fec_af99_0cc422f5c5ec.slice/crio-81ade3a1301175998b8abca1791797d3ba4253e2e4c033531ff653b8a5c2d34e WatchSource:0}: Error finding container 81ade3a1301175998b8abca1791797d3ba4253e2e4c033531ff653b8a5c2d34e: Status 404 returned error can't find the container with id 81ade3a1301175998b8abca1791797d3ba4253e2e4c033531ff653b8a5c2d34e
Oct 30 20:29:07.615498 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:07.615474    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"]
Oct 30 20:29:07.619142 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:29:07.619114    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454da90a_e4c7_43b2_9a74_4942d70cb222.slice/crio-233795cfe27a6d345bc8be7bfbc8a2a1ee52e04d1496cc2bcc306ce22ba7a743 WatchSource:0}: Error finding container 233795cfe27a6d345bc8be7bfbc8a2a1ee52e04d1496cc2bcc306ce22ba7a743: Status 404 returned error can't find the container with id 233795cfe27a6d345bc8be7bfbc8a2a1ee52e04d1496cc2bcc306ce22ba7a743
Oct 30 20:29:08.472182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.472149    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" event={"ID":"454da90a-e4c7-43b2-9a74-4942d70cb222","Type":"ContainerStarted","Data":"f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097"}
Oct 30 20:29:08.472609 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.472201    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" event={"ID":"454da90a-e4c7-43b2-9a74-4942d70cb222","Type":"ContainerStarted","Data":"233795cfe27a6d345bc8be7bfbc8a2a1ee52e04d1496cc2bcc306ce22ba7a743"}
Oct 30 20:29:08.472609 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.472247    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:08.473895 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.473860    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" event={"ID":"d20f3399-aa21-4fec-af99-0cc422f5c5ec","Type":"ContainerStarted","Data":"dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad"}
Oct 30 20:29:08.474073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.473906    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:08.474073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.473922    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" event={"ID":"d20f3399-aa21-4fec-af99-0cc422f5c5ec","Type":"ContainerStarted","Data":"81ade3a1301175998b8abca1791797d3ba4253e2e4c033531ff653b8a5c2d34e"}
Oct 30 20:29:08.492584 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.492541    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" podStartSLOduration=2.492529701 podStartE2EDuration="2.492529701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:08.491481124 +0000 UTC m=+1481.394901151" watchObservedRunningTime="2026-10-30 20:29:08.492529701 +0000 UTC m=+1481.395949727"
Oct 30 20:29:08.513112 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:08.513069    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" podStartSLOduration=2.513057303 podStartE2EDuration="2.513057303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:08.512363348 +0000 UTC m=+1481.415783375" watchObservedRunningTime="2026-10-30 20:29:08.513057303 +0000 UTC m=+1481.416477328"
Oct 30 20:29:09.274480 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:09.274437    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:29:19.483179 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:19.483149    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:29:19.483640 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:19.483587    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:29:19.657959 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:29:19.657922    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:29:24.321983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:24.321940    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:29:28.534166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.534129    6058 scope.go:122] "RemoveContainer" containerID="e793e9b23c5b7e052e1ea2814548839c08ed1fa933e85cbe32339209469941a7"
Oct 30 20:29:28.542631 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.542614    6058 scope.go:122] "RemoveContainer" containerID="d33133d9ffaa2e913024720364adc8660b25c8ec190ed7dfa8eea2f4098bc2ca"
Oct 30 20:29:28.550238 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.550216    6058 scope.go:122] "RemoveContainer" containerID="856f98045a316ad547acec9c10e9aadc0810994fd31fa98638553b656bb6a353"
Oct 30 20:29:28.558684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.558658    6058 scope.go:122] "RemoveContainer" containerID="ca73030547edf79878c76d9edfa10b48a3b3723565db2f1ffb74b9198a0e3fdd"
Oct 30 20:29:28.567243 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.567224    6058 scope.go:122] "RemoveContainer" containerID="d1c06661730d0ebe9ac775c3acaa0570fb84502d9002c8e32d46c48ac3a6932a"
Oct 30 20:29:28.575296 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.575264    6058 scope.go:122] "RemoveContainer" containerID="81a4603acd1e4860d6b6c73ad19666561c33105007099a9ec10fea066f200e43"
Oct 30 20:29:28.583241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:28.583224    6058 scope.go:122] "RemoveContainer" containerID="97b0a1322ea2d14c69ec418865223364260377bd3f4e4943626ffa236075fca9"
Oct 30 20:29:39.664909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:39.664872    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:29:43.124444 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.124411    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"]
Oct 30 20:29:43.129781 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.129762    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.137241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.136908    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"]
Oct 30 20:29:43.137241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.136957    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:29:43.137241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.137057    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:43.137241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.137064    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:29:43.137241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.137142    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:29:43.210623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.210596    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g747v\" (UniqueName: \"kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.210789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.210706    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.210789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.210748    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.311992 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.311962    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.312151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.312088    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.312303 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.312283    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g747v\" (UniqueName: \"kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.312757 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.312732    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.314403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.314379    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.325321 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.325300    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g747v\" (UniqueName: \"kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v\") pod \"maas-api-revoke-sub-3f44cd44-fwmlm\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.444800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.444750    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:43.721145 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:43.721123    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"]
Oct 30 20:29:43.723498 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:29:43.723471    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d988fcc_ac23_4fd8_bc93_c132ed17b327.slice/crio-3c8cc3b14c457bd05abcef7dbde8da604661287a8d4561c1655a38847006dede WatchSource:0}: Error finding container 3c8cc3b14c457bd05abcef7dbde8da604661287a8d4561c1655a38847006dede: Status 404 returned error can't find the container with id 3c8cc3b14c457bd05abcef7dbde8da604661287a8d4561c1655a38847006dede
Oct 30 20:29:44.667302 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:44.667272    6058 generic.go:356] "Generic (PLEG): container finished" podID="2d988fcc-ac23-4fd8-bc93-c132ed17b327" containerID="2092d0d9e37c8ab8a3e0b68289460810db593a7cefccdc65c814f3bda796ea7d" exitCode=0
Oct 30 20:29:44.667729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:44.667339    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm" event={"ID":"2d988fcc-ac23-4fd8-bc93-c132ed17b327","Type":"ContainerDied","Data":"2092d0d9e37c8ab8a3e0b68289460810db593a7cefccdc65c814f3bda796ea7d"}
Oct 30 20:29:44.667729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:44.667380    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm" event={"ID":"2d988fcc-ac23-4fd8-bc93-c132ed17b327","Type":"ContainerStarted","Data":"3c8cc3b14c457bd05abcef7dbde8da604661287a8d4561c1655a38847006dede"}
Oct 30 20:29:45.808108 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.808079    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:45.935554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.935478    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v\" (UniqueName: \"kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v\") pod \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") "
Oct 30 20:29:45.935702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.935589    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca\") pod \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") "
Oct 30 20:29:45.935702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.935623    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca\") pod \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\" (UID: \"2d988fcc-ac23-4fd8-bc93-c132ed17b327\") "
Oct 30 20:29:45.936039 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.935994    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca" pod "2d988fcc-ac23-4fd8-bc93-c132ed17b327" (UID: "2d988fcc-ac23-4fd8-bc93-c132ed17b327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:45.937554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.937519    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca" pod "2d988fcc-ac23-4fd8-bc93-c132ed17b327" (UID: "2d988fcc-ac23-4fd8-bc93-c132ed17b327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:45.937680 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:45.937608    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v" pod "2d988fcc-ac23-4fd8-bc93-c132ed17b327" (UID: "2d988fcc-ac23-4fd8-bc93-c132ed17b327"). InnerVolumeSpecName "kube-api-access-g747v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:46.037165 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:46.037141    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:46.037165 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:46.037163    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d988fcc-ac23-4fd8-bc93-c132ed17b327-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:46.037293 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:46.037172    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g747v\" (UniqueName: \"kubernetes.io/projected/2d988fcc-ac23-4fd8-bc93-c132ed17b327-kube-api-access-g747v\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:46.680230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:46.680196    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm" event={"ID":"2d988fcc-ac23-4fd8-bc93-c132ed17b327","Type":"ContainerDied","Data":"3c8cc3b14c457bd05abcef7dbde8da604661287a8d4561c1655a38847006dede"}
Oct 30 20:29:46.680432 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:46.680232    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"
Oct 30 20:29:46.680432 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:46.680246    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8cc3b14c457bd05abcef7dbde8da604661287a8d4561c1655a38847006dede"
Oct 30 20:29:49.655816 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:29:49.655780    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:29:54.061800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.061758    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:29:54.183158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.183125    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"]
Oct 30 20:29:54.183564 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.183550    6058 cpu_manager.go:414] "containerMap: removing container" podUID="2d988fcc-ac23-4fd8-bc93-c132ed17b327" containerName="revoke-keys"
Oct 30 20:29:54.183614 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.183565    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d988fcc-ac23-4fd8-bc93-c132ed17b327" containerName="revoke-keys"
Oct 30 20:29:54.183656 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.183628    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d988fcc-ac23-4fd8-bc93-c132ed17b327" containerName="revoke-keys"
Oct 30 20:29:54.187131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.187114    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.192047 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.192027    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:29:54.195698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.195675    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:29:54.195839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.195734    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:29:54.197510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.197451    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"]
Oct 30 20:29:54.212759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.212731    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2n9\" (UniqueName: \"kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.212872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.212785    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.212920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.212884    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.291605 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.291576    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:54.313873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.313811    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2n9\" (UniqueName: \"kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.313873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.313855    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.314032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.313902    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.314592 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.314564    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.316227 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.316206    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.326630 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.326605    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2n9\" (UniqueName: \"kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.501044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.500975    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:54.774929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:54.774889    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"]
Oct 30 20:29:54.777645 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:29:54.777601    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode011f897_7885_4212_bdd2_c8a725a7972b.slice/crio-e03b675c4f99d90ea7afecfd4a8be8545ab47c498da23d6f02d5de778ca7a1a8 WatchSource:0}: Error finding container e03b675c4f99d90ea7afecfd4a8be8545ab47c498da23d6f02d5de778ca7a1a8: Status 404 returned error can't find the container with id e03b675c4f99d90ea7afecfd4a8be8545ab47c498da23d6f02d5de778ca7a1a8
Oct 30 20:29:55.725769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:55.725671    6058 generic.go:356] "Generic (PLEG): container finished" podID="e011f897-7885-4212-bdd2-c8a725a7972b" containerID="e6cf85daa313ae2722b3c54f58256b36268b8cb38b62796b57809477e085f59b" exitCode=0
Oct 30 20:29:55.725769 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:55.725754    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j" event={"ID":"e011f897-7885-4212-bdd2-c8a725a7972b","Type":"ContainerDied","Data":"e6cf85daa313ae2722b3c54f58256b36268b8cb38b62796b57809477e085f59b"}
Oct 30 20:29:55.726182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:55.725804    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j" event={"ID":"e011f897-7885-4212-bdd2-c8a725a7972b","Type":"ContainerStarted","Data":"e03b675c4f99d90ea7afecfd4a8be8545ab47c498da23d6f02d5de778ca7a1a8"}
Oct 30 20:29:56.861059 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.861031    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:56.935818 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.935783    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca\") pod \"e011f897-7885-4212-bdd2-c8a725a7972b\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") "
Oct 30 20:29:56.935986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.935969    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca\") pod \"e011f897-7885-4212-bdd2-c8a725a7972b\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") "
Oct 30 20:29:56.936062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.936019    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9\" (UniqueName: \"kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9\") pod \"e011f897-7885-4212-bdd2-c8a725a7972b\" (UID: \"e011f897-7885-4212-bdd2-c8a725a7972b\") "
Oct 30 20:29:56.936322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.936295    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca" pod "e011f897-7885-4212-bdd2-c8a725a7972b" (UID: "e011f897-7885-4212-bdd2-c8a725a7972b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:56.937854 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.937833    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca" pod "e011f897-7885-4212-bdd2-c8a725a7972b" (UID: "e011f897-7885-4212-bdd2-c8a725a7972b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:56.937969 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:56.937948    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9" pod "e011f897-7885-4212-bdd2-c8a725a7972b" (UID: "e011f897-7885-4212-bdd2-c8a725a7972b"). InnerVolumeSpecName "kube-api-access-nq2n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:57.037523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:57.037450    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:57.037523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:57.037486    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e011f897-7885-4212-bdd2-c8a725a7972b-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:57.037523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:57.037501    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2n9\" (UniqueName: \"kubernetes.io/projected/e011f897-7885-4212-bdd2-c8a725a7972b-kube-api-access-nq2n9\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:57.737133 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:57.737103    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"
Oct 30 20:29:57.737308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:57.737145    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j" event={"ID":"e011f897-7885-4212-bdd2-c8a725a7972b","Type":"ContainerDied","Data":"e03b675c4f99d90ea7afecfd4a8be8545ab47c498da23d6f02d5de778ca7a1a8"}
Oct 30 20:29:57.737308 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:29:57.737218    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03b675c4f99d90ea7afecfd4a8be8545ab47c498da23d6f02d5de778ca7a1a8"
Oct 30 20:30:00.135129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.135098    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"]
Oct 30 20:30:00.135508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.135495    6058 cpu_manager.go:414] "containerMap: removing container" podUID="e011f897-7885-4212-bdd2-c8a725a7972b" containerName="revoke-keys"
Oct 30 20:30:00.135552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.135509    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e011f897-7885-4212-bdd2-c8a725a7972b" containerName="revoke-keys"
Oct 30 20:30:00.135588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.135568    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e011f897-7885-4212-bdd2-c8a725a7972b" containerName="revoke-keys"
Oct 30 20:30:00.139022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.138983    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.145063 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.145040    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:00.145252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.145237    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:30:00.145303 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.145256    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:00.148719 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.148698    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"]
Oct 30 20:30:00.169913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.165239    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbxq\" (UniqueName: \"kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq\") pod \"maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.169913 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.165295    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.265792 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.265765    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.265920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.265873    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbxq\" (UniqueName: \"kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq\") pod \"maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.266412 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.266389    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.279102 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.279078    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbxq\" (UniqueName: \"kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq\") pod \"maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.453490 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.453436    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:00.729813 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.729787    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"]
Oct 30 20:30:00.732131 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:30:00.732103    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5297ce_cce2_4a7a_8695_57e15d8ca7b2.slice/crio-562c4804f4ac52f857adf9e99499c1bcf3010f388cf44f76609402962f0d0cba WatchSource:0}: Error finding container 562c4804f4ac52f857adf9e99499c1bcf3010f388cf44f76609402962f0d0cba: Status 404 returned error can't find the container with id 562c4804f4ac52f857adf9e99499c1bcf3010f388cf44f76609402962f0d0cba
Oct 30 20:30:00.752337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:00.752309    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x" event={"ID":"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2","Type":"ContainerStarted","Data":"562c4804f4ac52f857adf9e99499c1bcf3010f388cf44f76609402962f0d0cba"}
Oct 30 20:30:01.758647 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:01.758614    6058 generic.go:356] "Generic (PLEG): container finished" podID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerID="b96d61a9ea38006ee17e407fa8dea37640bf0c6717a74c8dcaaeeb39e666779c" exitCode=60
Oct 30 20:30:01.758975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:01.758679    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x" event={"ID":"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2","Type":"ContainerDied","Data":"b96d61a9ea38006ee17e407fa8dea37640bf0c6717a74c8dcaaeeb39e666779c"}
Oct 30 20:30:01.758975 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:01.758897    6058 scope.go:122] "RemoveContainer" containerID="b96d61a9ea38006ee17e407fa8dea37640bf0c6717a74c8dcaaeeb39e666779c"
Oct 30 20:30:02.764875 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:02.764801    6058 generic.go:356] "Generic (PLEG): container finished" podID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerID="6a078d712918fef1b4ca3e2ceba32c188efbb0e48bd076604d760e008571be9f" exitCode=60
Oct 30 20:30:02.764875 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:02.764868    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x" event={"ID":"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2","Type":"ContainerDied","Data":"6a078d712918fef1b4ca3e2ceba32c188efbb0e48bd076604d760e008571be9f"}
Oct 30 20:30:02.765359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:02.764960    6058 scope.go:122] "RemoveContainer" containerID="b96d61a9ea38006ee17e407fa8dea37640bf0c6717a74c8dcaaeeb39e666779c"
Oct 30 20:30:02.765359 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:02.765158    6058 scope.go:122] "RemoveContainer" containerID="6a078d712918fef1b4ca3e2ceba32c188efbb0e48bd076604d760e008571be9f"
Oct 30 20:30:02.765506 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:02.765377    6058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x_odh-ai-gateway-infra(5d5297ce-cce2-4a7a-8695-57e15d8ca7b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2"
Oct 30 20:30:03.770840 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:03.770813    6058 scope.go:122] "RemoveContainer" containerID="6a078d712918fef1b4ca3e2ceba32c188efbb0e48bd076604d760e008571be9f"
Oct 30 20:30:03.771243 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:03.771027    6058 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x_odh-ai-gateway-infra(5d5297ce-cce2-4a7a-8695-57e15d8ca7b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2"
Oct 30 20:30:04.335117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:04.335080    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"]
Oct 30 20:30:04.348486 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:04.348458    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"]
Oct 30 20:30:05.220794 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.220760    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"]
Oct 30 20:30:05.358284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.358252    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:05.413930 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.413895    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"]
Oct 30 20:30:05.415702 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.415679    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-402199-zbx5j"]
Oct 30 20:30:05.419398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.419378    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq\" (UniqueName: \"kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq\") pod \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") "
Oct 30 20:30:05.419518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.419504    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca\") pod \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\" (UID: \"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2\") "
Oct 30 20:30:05.419874 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.419856    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca" pod "5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" (UID: "5d5297ce-cce2-4a7a-8695-57e15d8ca7b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:05.421388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.421365    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq" pod "5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" (UID: "5d5297ce-cce2-4a7a-8695-57e15d8ca7b2"). InnerVolumeSpecName "kube-api-access-qwbxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:05.521337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.521240    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-maas-api-service-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:05.521337 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.521283    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwbxq\" (UniqueName: \"kubernetes.io/projected/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2-kube-api-access-qwbxq\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:05.787550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.787466    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"
Oct 30 20:30:05.787698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.787466    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x" event={"ID":"5d5297ce-cce2-4a7a-8695-57e15d8ca7b2","Type":"ContainerDied","Data":"562c4804f4ac52f857adf9e99499c1bcf3010f388cf44f76609402962f0d0cba"}
Oct 30 20:30:05.787698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.787627    6058 scope.go:122] "RemoveContainer" containerID="6a078d712918fef1b4ca3e2ceba32c188efbb0e48bd076604d760e008571be9f"
Oct 30 20:30:05.811131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.811101    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"]
Oct 30 20:30:05.814486 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:05.814459    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-402199-2988987d6p2x"]
Oct 30 20:30:07.029445 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:07.029402    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:30:07.211800 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:07.211770    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" path="/var/lib/kubelet/pods/5d5297ce-cce2-4a7a-8695-57e15d8ca7b2/volumes"
Oct 30 20:30:07.212215 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:07.212200    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e011f897-7885-4212-bdd2-c8a725a7972b" path="/var/lib/kubelet/pods/e011f897-7885-4212-bdd2-c8a725a7972b/volumes"
Oct 30 20:30:09.335681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:09.335581    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing" containerID="cri-o://f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097" gracePeriod=25
Oct 30 20:30:09.348729 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:09.348693    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing" containerID="cri-o://dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad" gracePeriod=25
Oct 30 20:30:09.481714 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:09.481665    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:09.481714 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:09.481704    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:13.743323 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.743288    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"]
Oct 30 20:30:13.748905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.748877    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2bs7p"]
Oct 30 20:30:13.770189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.770159    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"]
Oct 30 20:30:13.770582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.770570    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerName="cleanup"
Oct 30 20:30:13.770624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.770583    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerName="cleanup"
Oct 30 20:30:13.770658 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.770653    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerName="cleanup"
Oct 30 20:30:13.770697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.770664    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerName="cleanup"
Oct 30 20:30:13.775023 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.774996    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:13.782272 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.782233    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:13.782802 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.782780    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:30:13.782928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.782802    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:13.782928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.782910    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:13.783476 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.783456    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"]
Oct 30 20:30:13.902031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.901969    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkfx\" (UniqueName: \"kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:13.902204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.902107    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:13.902204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:13.902165    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.003700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.003613    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.003871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.003761    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkfx\" (UniqueName: \"kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.003871 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.003820    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.004487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.004457    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.007075 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.007055    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.016638 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.016613    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkfx\" (UniqueName: \"kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx\") pod \"maas-api-revoke-sub-3d4156f1-x6hgl\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.089567 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.089535    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:14.368993 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.368964    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"]
Oct 30 20:30:14.372481 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:30:14.372451    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc01c4a2_9a5a_406e_847c_2bd770d09fec.slice/crio-9a1518492a05e33f5820d2bac3d6f3c001aa5290116d5c40aaf690fc62ce151d WatchSource:0}: Error finding container 9a1518492a05e33f5820d2bac3d6f3c001aa5290116d5c40aaf690fc62ce151d: Status 404 returned error can't find the container with id 9a1518492a05e33f5820d2bac3d6f3c001aa5290116d5c40aaf690fc62ce151d
Oct 30 20:30:14.839593 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.839553    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl" event={"ID":"cc01c4a2-9a5a-406e-847c-2bd770d09fec","Type":"ContainerStarted","Data":"807aca43892ff542424f6f8aa69bf2e79430875a4b01dd3ee45d35f812324d2e"}
Oct 30 20:30:14.839947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.839617    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl" event={"ID":"cc01c4a2-9a5a-406e-847c-2bd770d09fec","Type":"ContainerStarted","Data":"9a1518492a05e33f5820d2bac3d6f3c001aa5290116d5c40aaf690fc62ce151d"}
Oct 30 20:30:14.859742 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:14.859696    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl" podStartSLOduration=1.8596851079999999 podStartE2EDuration="1.859685108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:14.857825744 +0000 UTC m=+1547.761245766" watchObservedRunningTime="2026-10-30 20:30:14.859685108 +0000 UTC m=+1547.763105133"
Oct 30 20:30:15.211232 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:15.211204    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251" path="/var/lib/kubelet/pods/0bd3fcd4-24cd-41f2-bed9-0b5d3e7e0251/volumes"
Oct 30 20:30:15.846461 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:15.846432    6058 generic.go:356] "Generic (PLEG): container finished" podID="cc01c4a2-9a5a-406e-847c-2bd770d09fec" containerID="807aca43892ff542424f6f8aa69bf2e79430875a4b01dd3ee45d35f812324d2e" exitCode=0
Oct 30 20:30:15.846830 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:15.846476    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl" event={"ID":"cc01c4a2-9a5a-406e-847c-2bd770d09fec","Type":"ContainerDied","Data":"807aca43892ff542424f6f8aa69bf2e79430875a4b01dd3ee45d35f812324d2e"}
Oct 30 20:30:16.982798 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:16.982768    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:17.132448 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.132418    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx\" (UniqueName: \"kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx\") pod \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") "
Oct 30 20:30:17.132596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.132458    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca\") pod \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") "
Oct 30 20:30:17.132596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.132485    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca\") pod \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\" (UID: \"cc01c4a2-9a5a-406e-847c-2bd770d09fec\") "
Oct 30 20:30:17.132846 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.132817    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca" pod "cc01c4a2-9a5a-406e-847c-2bd770d09fec" (UID: "cc01c4a2-9a5a-406e-847c-2bd770d09fec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:17.134487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.134468    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx" pod "cc01c4a2-9a5a-406e-847c-2bd770d09fec" (UID: "cc01c4a2-9a5a-406e-847c-2bd770d09fec"). InnerVolumeSpecName "kube-api-access-9dkfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:17.134578 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.134509    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca" pod "cc01c4a2-9a5a-406e-847c-2bd770d09fec" (UID: "cc01c4a2-9a5a-406e-847c-2bd770d09fec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:17.233276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.233250    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkfx\" (UniqueName: \"kubernetes.io/projected/cc01c4a2-9a5a-406e-847c-2bd770d09fec-kube-api-access-9dkfx\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.233276 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.233272    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.233410 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.233282    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc01c4a2-9a5a-406e-847c-2bd770d09fec-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.858100 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.858074    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"
Oct 30 20:30:17.858261 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.858063    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl" event={"ID":"cc01c4a2-9a5a-406e-847c-2bd770d09fec","Type":"ContainerDied","Data":"9a1518492a05e33f5820d2bac3d6f3c001aa5290116d5c40aaf690fc62ce151d"}
Oct 30 20:30:17.858261 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:17.858218    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1518492a05e33f5820d2bac3d6f3c001aa5290116d5c40aaf690fc62ce151d"
Oct 30 20:30:19.482154 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:19.482107    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:19.482550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:19.482205    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:19.654785 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:19.654752    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:30:25.152567 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:25.152525    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:30:27.205146 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:27.205107    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:27.219165 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:27.219124    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:27.232132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:27.232101    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:27.232347 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:27.232322    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:28.628273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:28.628239    6058 scope.go:122] "RemoveContainer" containerID="567423996aaea005ae2bfbd4669c1a3ea02de4a126d56ce1d3934b6162455797"
Oct 30 20:30:28.636695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:28.636673    6058 scope.go:122] "RemoveContainer" containerID="70ac6e82366780f6d1e469c6fb7a73b53ef9888269d96cd971d603e375789ca9"
Oct 30 20:30:29.361805 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.361762    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:30:29.481585 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:30:29.481560    6058 logging.go:55] [core] [Channel #628 SubChannel #630]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.121:50052", ServerName: "10.133.0.121:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.121:50052: connect: connection refused"
Oct 30 20:30:29.481744 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:30:29.481561    6058 logging.go:55] [core] [Channel #627 SubChannel #629]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.122:50052", ServerName: "10.133.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.122:50052: connect: connection refused"
Oct 30 20:30:29.603246 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.603177    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:30:29.641523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.641498    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:30:29.665366 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.665337    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468\" (UniqueName: \"kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468\") pod \"454da90a-e4c7-43b2-9a74-4942d70cb222\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") "
Oct 30 20:30:29.665525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.665384    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k\" (UniqueName: \"kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k\") pod \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") "
Oct 30 20:30:29.665525 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.665435    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume\") pod \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\" (UID: \"d20f3399-aa21-4fec-af99-0cc422f5c5ec\") "
Oct 30 20:30:29.665652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.665553    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume\") pod \"454da90a-e4c7-43b2-9a74-4942d70cb222\" (UID: \"454da90a-e4c7-43b2-9a74-4942d70cb222\") "
Oct 30 20:30:29.666240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.666214    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume" pod "454da90a-e4c7-43b2-9a74-4942d70cb222" (UID: "454da90a-e4c7-43b2-9a74-4942d70cb222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:29.666772 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.666748    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume" pod "d20f3399-aa21-4fec-af99-0cc422f5c5ec" (UID: "d20f3399-aa21-4fec-af99-0cc422f5c5ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:29.668381 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.668351    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k" pod "d20f3399-aa21-4fec-af99-0cc422f5c5ec" (UID: "d20f3399-aa21-4fec-af99-0cc422f5c5ec"). InnerVolumeSpecName "kube-api-access-96d7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:29.668481 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.668421    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468" pod "454da90a-e4c7-43b2-9a74-4942d70cb222" (UID: "454da90a-e4c7-43b2-9a74-4942d70cb222"). InnerVolumeSpecName "kube-api-access-4x468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:29.766883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.766848    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454da90a-e4c7-43b2-9a74-4942d70cb222-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:29.766883 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.766881    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x468\" (UniqueName: \"kubernetes.io/projected/454da90a-e4c7-43b2-9a74-4942d70cb222-kube-api-access-4x468\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:29.767076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.766896    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96d7k\" (UniqueName: \"kubernetes.io/projected/d20f3399-aa21-4fec-af99-0cc422f5c5ec-kube-api-access-96d7k\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:29.767076 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.766910    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20f3399-aa21-4fec-af99-0cc422f5c5ec-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:29.940132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.940096    6058 generic.go:356] "Generic (PLEG): container finished" podID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerID="f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097" exitCode=0
Oct 30 20:30:29.940292 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.940159    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"
Oct 30 20:30:29.940292 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.940178    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" event={"ID":"454da90a-e4c7-43b2-9a74-4942d70cb222","Type":"ContainerDied","Data":"f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097"}
Oct 30 20:30:29.940292 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.940267    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" event={"ID":"454da90a-e4c7-43b2-9a74-4942d70cb222","Type":"ContainerDied","Data":"233795cfe27a6d345bc8be7bfbc8a2a1ee52e04d1496cc2bcc306ce22ba7a743"}
Oct 30 20:30:29.940452 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.940301    6058 scope.go:122] "RemoveContainer" containerID="f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097"
Oct 30 20:30:29.941652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.941527    6058 generic.go:356] "Generic (PLEG): container finished" podID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerID="dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad" exitCode=0
Oct 30 20:30:29.941652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.941558    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" event={"ID":"d20f3399-aa21-4fec-af99-0cc422f5c5ec","Type":"ContainerDied","Data":"dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad"}
Oct 30 20:30:29.941652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.941581    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"
Oct 30 20:30:29.941652 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.941600    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" event={"ID":"d20f3399-aa21-4fec-af99-0cc422f5c5ec","Type":"ContainerDied","Data":"81ade3a1301175998b8abca1791797d3ba4253e2e4c033531ff653b8a5c2d34e"}
Oct 30 20:30:29.950487 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.950473    6058 scope.go:122] "RemoveContainer" containerID="f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097"
Oct 30 20:30:29.950752 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:29.950734    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097\": container with ID starting with f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097 not found: ID does not exist" containerID="f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097"
Oct 30 20:30:29.950799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.950758    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097"} err="failed to get container status \"f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097\": rpc error: code = NotFound desc = could not find container \"f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097\": container with ID starting with f11d5352e29d67f5d55fc975423f3a21bb7e1c49dcb5d93b2422c05d69a98097 not found: ID does not exist"
Oct 30 20:30:29.950799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.950775    6058 scope.go:122] "RemoveContainer" containerID="dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad"
Oct 30 20:30:29.958369 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.958349    6058 scope.go:122] "RemoveContainer" containerID="dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad"
Oct 30 20:30:29.958656 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:29.958638    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad\": container with ID starting with dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad not found: ID does not exist" containerID="dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad"
Oct 30 20:30:29.958706 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.958664    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad"} err="failed to get container status \"dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad\": rpc error: code = NotFound desc = could not find container \"dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad\": container with ID starting with dd252eb32f14ddf16f483d028dc1d496d00488e5128be1a2c6e686e9f0f9afad not found: ID does not exist"
Oct 30 20:30:29.965986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.965968    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"]
Oct 30 20:30:29.971940 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.971919    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f"]
Oct 30 20:30:29.984563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.984541    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"]
Oct 30 20:30:29.989260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:29.989243    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b"]
Oct 30 20:30:30.481678 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:30.481644    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-402199-84f6csj79b" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.121:50052\" within 1s: context deadline exceeded"
Oct 30 20:30:30.481859 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:30.481638    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-402199-57c8465b7bfs6f" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.122:50052\" within 1s: context deadline exceeded"
Oct 30 20:30:31.212018 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:31.211974    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" path="/var/lib/kubelet/pods/454da90a-e4c7-43b2-9a74-4942d70cb222/volumes"
Oct 30 20:30:31.212407 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:31.212391    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" path="/var/lib/kubelet/pods/d20f3399-aa21-4fec-af99-0cc422f5c5ec/volumes"
Oct 30 20:30:32.520641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.520604    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"]
Oct 30 20:30:32.521063 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.520797    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p" podUID="ae638cf2-46c5-4c5c-a383-ae889399a8a6" containerName="manager" containerID="cri-o://2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4" gracePeriod=10
Oct 30 20:30:32.770118 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.770095    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:30:32.792040 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.791958    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d\" (UniqueName: \"kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d\") pod \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") "
Oct 30 20:30:32.792170 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.792082    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume\") pod \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\" (UID: \"ae638cf2-46c5-4c5c-a383-ae889399a8a6\") "
Oct 30 20:30:32.792416 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.792392    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume" pod "ae638cf2-46c5-4c5c-a383-ae889399a8a6" (UID: "ae638cf2-46c5-4c5c-a383-ae889399a8a6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:32.794355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.794315    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d" pod "ae638cf2-46c5-4c5c-a383-ae889399a8a6" (UID: "ae638cf2-46c5-4c5c-a383-ae889399a8a6"). InnerVolumeSpecName "kube-api-access-6vf8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:32.892839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.892807    6058 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae638cf2-46c5-4c5c-a383-ae889399a8a6-extensions-socket-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:32.892839 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.892834    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vf8d\" (UniqueName: \"kubernetes.io/projected/ae638cf2-46c5-4c5c-a383-ae889399a8a6-kube-api-access-6vf8d\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:32.959979 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.959945    6058 generic.go:356] "Generic (PLEG): container finished" podID="ae638cf2-46c5-4c5c-a383-ae889399a8a6" containerID="2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4" exitCode=0
Oct 30 20:30:32.960169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.959991    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"
Oct 30 20:30:32.960169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.960035    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p" event={"ID":"ae638cf2-46c5-4c5c-a383-ae889399a8a6","Type":"ContainerDied","Data":"2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4"}
Oct 30 20:30:32.960169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.960098    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p" event={"ID":"ae638cf2-46c5-4c5c-a383-ae889399a8a6","Type":"ContainerDied","Data":"a6c7e107b381f8b4009f3f272fd4554420141c9e43f3284828a6d35261a275dd"}
Oct 30 20:30:32.960169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.960130    6058 scope.go:122] "RemoveContainer" containerID="2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4"
Oct 30 20:30:32.976792 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.976758    6058 scope.go:122] "RemoveContainer" containerID="2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4"
Oct 30 20:30:32.977093 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:32.977067    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4\": container with ID starting with 2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4 not found: ID does not exist" containerID="2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4"
Oct 30 20:30:32.977151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.977092    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4"} err="failed to get container status \"2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4\": rpc error: code = NotFound desc = could not find container \"2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4\": container with ID starting with 2de0beb5e67084d20749d8227dc59582123c4384581b054207c5dfb1f4ccbdf4 not found: ID does not exist"
Oct 30 20:30:32.985998 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.985977    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"]
Oct 30 20:30:32.993881 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:32.993863    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-s2x2p"]
Oct 30 20:30:33.210716 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:30:33.210681    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae638cf2-46c5-4c5c-a383-ae889399a8a6" path="/var/lib/kubelet/pods/ae638cf2-46c5-4c5c-a383-ae889399a8a6/volumes"
Oct 30 20:30:49.656255 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:30:49.656213    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:31:19.654943 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:31:19.654909    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:31:33.657300 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657264    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"]
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657606    6058 cpu_manager.go:414] "containerMap: removing container" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657617    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657633    6058 cpu_manager.go:414] "containerMap: removing container" podUID="ae638cf2-46c5-4c5c-a383-ae889399a8a6" containerName="manager"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657638    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae638cf2-46c5-4c5c-a383-ae889399a8a6" containerName="manager"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657648    6058 cpu_manager.go:414] "containerMap: removing container" podUID="cc01c4a2-9a5a-406e-847c-2bd770d09fec" containerName="revoke-keys"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657654    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc01c4a2-9a5a-406e-847c-2bd770d09fec" containerName="revoke-keys"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657671    6058 cpu_manager.go:414] "containerMap: removing container" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerName="cleanup"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657675    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5297ce-cce2-4a7a-8695-57e15d8ca7b2" containerName="cleanup"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657691    6058 cpu_manager.go:414] "containerMap: removing container" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing"
Oct 30 20:31:33.657701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657697    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing"
Oct 30 20:31:33.658035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657747    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae638cf2-46c5-4c5c-a383-ae889399a8a6" containerName="manager"
Oct 30 20:31:33.658035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657755    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc01c4a2-9a5a-406e-847c-2bd770d09fec" containerName="revoke-keys"
Oct 30 20:31:33.658035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657771    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="454da90a-e4c7-43b2-9a74-4942d70cb222" containerName="payload-processing"
Oct 30 20:31:33.658035 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.657778    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20f3399-aa21-4fec-af99-0cc422f5c5ec" containerName="payload-pre-processing"
Oct 30 20:31:33.662087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.662069    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.678314 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.678294    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t8dtz\""
Oct 30 20:31:33.684907 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.684883    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"]
Oct 30 20:31:33.733447 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.733420    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10ba5c8d-a75a-42e0-a5f1-57bb3abf312e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j6tq5\" (UID: \"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.733550 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.733464    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hsjl\" (UniqueName: \"kubernetes.io/projected/10ba5c8d-a75a-42e0-a5f1-57bb3abf312e-kube-api-access-9hsjl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j6tq5\" (UID: \"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.834672 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.834646    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10ba5c8d-a75a-42e0-a5f1-57bb3abf312e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j6tq5\" (UID: \"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.834861 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.834690    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsjl\" (UniqueName: \"kubernetes.io/projected/10ba5c8d-a75a-42e0-a5f1-57bb3abf312e-kube-api-access-9hsjl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j6tq5\" (UID: \"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.835123 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.835103    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10ba5c8d-a75a-42e0-a5f1-57bb3abf312e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j6tq5\" (UID: \"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.849001 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.848974    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsjl\" (UniqueName: \"kubernetes.io/projected/10ba5c8d-a75a-42e0-a5f1-57bb3abf312e-kube-api-access-9hsjl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j6tq5\" (UID: \"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:33.974994 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:33.974930    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:34.255759 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:34.255736    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"]
Oct 30 20:31:34.258429 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:31:34.258401    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ba5c8d_a75a_42e0_a5f1_57bb3abf312e.slice/crio-00ee10516ebd55e766f14c44c0bbe9e93323e1a9a4a49b07ed0853e3ac597b49 WatchSource:0}: Error finding container 00ee10516ebd55e766f14c44c0bbe9e93323e1a9a4a49b07ed0853e3ac597b49: Status 404 returned error can't find the container with id 00ee10516ebd55e766f14c44c0bbe9e93323e1a9a4a49b07ed0853e3ac597b49
Oct 30 20:31:35.243335 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:35.243301    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5" event={"ID":"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e","Type":"ContainerStarted","Data":"f315f955dcbda197b06e519d31f182da0d133edbc16d53e5c8638e07b8114faa"}
Oct 30 20:31:35.243732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:35.243356    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5" event={"ID":"10ba5c8d-a75a-42e0-a5f1-57bb3abf312e","Type":"ContainerStarted","Data":"00ee10516ebd55e766f14c44c0bbe9e93323e1a9a4a49b07ed0853e3ac597b49"}
Oct 30 20:31:35.243732 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:35.243397    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:35.271867 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:35.271817    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5" podStartSLOduration=2.271806213 podStartE2EDuration="2.271806213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:35.271307287 +0000 UTC m=+1628.174727314" watchObservedRunningTime="2026-10-30 20:31:35.271806213 +0000 UTC m=+1628.175226239"
Oct 30 20:31:46.250829 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:46.250752    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j6tq5"
Oct 30 20:31:49.655893 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:31:49.655857    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:31:51.845738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:51.845703    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:31:56.004048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.003996    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"]
Oct 30 20:31:56.007682 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.007666    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.017151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.017119    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:31:56.017273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.017161    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:31:56.017273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.017198    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:31:56.017273 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.017271    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:56.017440 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.017317    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"]
Oct 30 20:31:56.045014 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.044986    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.045130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.045064    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppfw2\" (UniqueName: \"kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.045130 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.045086    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.145860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.145818    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.145999 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.145940    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppfw2\" (UniqueName: \"kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.145999 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.145973    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.146426 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.146405    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.148275 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.148254    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.159171 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.159148    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppfw2\" (UniqueName: \"kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2\") pod \"maas-api-revoke-sub-d576e68b-xdxwt\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.320996 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.320937    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:56.596802 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:56.596718    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"]
Oct 30 20:31:56.600588 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:31:56.600552    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046f5cc6_ad54_4852_b1a3_b661bed25c6d.slice/crio-c60723eb325ffb32fee559a901b7409911c02ba02a77ddd2b1e8a5c80be1a072 WatchSource:0}: Error finding container c60723eb325ffb32fee559a901b7409911c02ba02a77ddd2b1e8a5c80be1a072: Status 404 returned error can't find the container with id c60723eb325ffb32fee559a901b7409911c02ba02a77ddd2b1e8a5c80be1a072
Oct 30 20:31:57.364027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:57.363972    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt" event={"ID":"046f5cc6-ad54-4852-b1a3-b661bed25c6d","Type":"ContainerStarted","Data":"ca2cdb5db63f828d56dfec22dc5cd7da12d5c3e9f076183c82a4883525629623"}
Oct 30 20:31:57.364403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:57.364047    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt" event={"ID":"046f5cc6-ad54-4852-b1a3-b661bed25c6d","Type":"ContainerStarted","Data":"c60723eb325ffb32fee559a901b7409911c02ba02a77ddd2b1e8a5c80be1a072"}
Oct 30 20:31:57.384297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:57.384253    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt" podStartSLOduration=2.384238478 podStartE2EDuration="2.384238478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:57.383091093 +0000 UTC m=+1650.286511116" watchObservedRunningTime="2026-10-30 20:31:57.384238478 +0000 UTC m=+1650.287658504"
Oct 30 20:31:58.369556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:58.369527    6058 generic.go:356] "Generic (PLEG): container finished" podID="046f5cc6-ad54-4852-b1a3-b661bed25c6d" containerID="ca2cdb5db63f828d56dfec22dc5cd7da12d5c3e9f076183c82a4883525629623" exitCode=0
Oct 30 20:31:58.369944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:58.369620    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt" event={"ID":"046f5cc6-ad54-4852-b1a3-b661bed25c6d","Type":"ContainerDied","Data":"ca2cdb5db63f828d56dfec22dc5cd7da12d5c3e9f076183c82a4883525629623"}
Oct 30 20:31:59.507679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.507656    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:31:59.567784 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.567751    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2\" (UniqueName: \"kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2\") pod \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") "
Oct 30 20:31:59.567962 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.567801    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca\") pod \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") "
Oct 30 20:31:59.567962 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.567846    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca\") pod \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\" (UID: \"046f5cc6-ad54-4852-b1a3-b661bed25c6d\") "
Oct 30 20:31:59.568250 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.568223    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca" pod "046f5cc6-ad54-4852-b1a3-b661bed25c6d" (UID: "046f5cc6-ad54-4852-b1a3-b661bed25c6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:59.569922 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.569892    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca" pod "046f5cc6-ad54-4852-b1a3-b661bed25c6d" (UID: "046f5cc6-ad54-4852-b1a3-b661bed25c6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:59.570036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.569949    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2" pod "046f5cc6-ad54-4852-b1a3-b661bed25c6d" (UID: "046f5cc6-ad54-4852-b1a3-b661bed25c6d"). InnerVolumeSpecName "kube-api-access-ppfw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:59.669524 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.669455    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppfw2\" (UniqueName: \"kubernetes.io/projected/046f5cc6-ad54-4852-b1a3-b661bed25c6d-kube-api-access-ppfw2\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:31:59.669524 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.669481    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:31:59.669524 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:31:59.669492    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046f5cc6-ad54-4852-b1a3-b661bed25c6d-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:00.379620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:00.379588    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"
Oct 30 20:32:00.381705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:00.379620    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt" event={"ID":"046f5cc6-ad54-4852-b1a3-b661bed25c6d","Type":"ContainerDied","Data":"c60723eb325ffb32fee559a901b7409911c02ba02a77ddd2b1e8a5c80be1a072"}
Oct 30 20:32:00.381705 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:00.379662    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c60723eb325ffb32fee559a901b7409911c02ba02a77ddd2b1e8a5c80be1a072"
Oct 30 20:32:06.769806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:06.769756    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:32:12.102229 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.102185    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"]
Oct 30 20:32:12.102622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.102586    6058 cpu_manager.go:414] "containerMap: removing container" podUID="046f5cc6-ad54-4852-b1a3-b661bed25c6d" containerName="revoke-keys"
Oct 30 20:32:12.102622 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.102597    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046f5cc6-ad54-4852-b1a3-b661bed25c6d" containerName="revoke-keys"
Oct 30 20:32:12.102693 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.102663    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="046f5cc6-ad54-4852-b1a3-b661bed25c6d" containerName="revoke-keys"
Oct 30 20:32:12.105905 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.105888    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.110523 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.110500    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:32:12.111561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.111536    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:32:12.111561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.111555    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:32:12.111752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.111606    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:32:12.116603 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.116583    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"]
Oct 30 20:32:12.228877 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.228842    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnvn\" (UniqueName: \"kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.229100 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.228969    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.229100 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.229023    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.330162 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.330133    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.330338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.330165    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.330338 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.330221    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnvn\" (UniqueName: \"kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.330853 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.330827    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.332506 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.332477    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.343282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.343262    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnvn\" (UniqueName: \"kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn\") pod \"maas-api-revoke-sub-d2bc93bc-xskkc\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.420918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.420896    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:12.697554 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:12.697425    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"]
Oct 30 20:32:12.700169 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:32:12.700138    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599a2da6_8cd0_4553_a4cf_98d64f120a45.slice/crio-0bc17d1e02599d7c237ef262fd072cb42a3f5496cb8a8a06a8d5b696dc39889e WatchSource:0}: Error finding container 0bc17d1e02599d7c237ef262fd072cb42a3f5496cb8a8a06a8d5b696dc39889e: Status 404 returned error can't find the container with id 0bc17d1e02599d7c237ef262fd072cb42a3f5496cb8a8a06a8d5b696dc39889e
Oct 30 20:32:13.445636 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:13.445595    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc" event={"ID":"599a2da6-8cd0-4553-a4cf-98d64f120a45","Type":"ContainerStarted","Data":"4d93737b8b0589169e4d51821137371bd29af8f7baeb31570e97c989413a5551"}
Oct 30 20:32:13.446147 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:13.445651    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc" event={"ID":"599a2da6-8cd0-4553-a4cf-98d64f120a45","Type":"ContainerStarted","Data":"0bc17d1e02599d7c237ef262fd072cb42a3f5496cb8a8a06a8d5b696dc39889e"}
Oct 30 20:32:13.466022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:13.465964    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc" podStartSLOduration=1.465951255 podStartE2EDuration="1.465951255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:13.464074692 +0000 UTC m=+1666.367494718" watchObservedRunningTime="2026-10-30 20:32:13.465951255 +0000 UTC m=+1666.369371280"
Oct 30 20:32:14.451548 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:14.451510    6058 generic.go:356] "Generic (PLEG): container finished" podID="599a2da6-8cd0-4553-a4cf-98d64f120a45" containerID="4d93737b8b0589169e4d51821137371bd29af8f7baeb31570e97c989413a5551" exitCode=0
Oct 30 20:32:14.451972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:14.451560    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc" event={"ID":"599a2da6-8cd0-4553-a4cf-98d64f120a45","Type":"ContainerDied","Data":"4d93737b8b0589169e4d51821137371bd29af8f7baeb31570e97c989413a5551"}
Oct 30 20:32:15.585228 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.585203    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:15.761947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.761884    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn\" (UniqueName: \"kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn\") pod \"599a2da6-8cd0-4553-a4cf-98d64f120a45\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") "
Oct 30 20:32:15.761947 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.761946    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca\") pod \"599a2da6-8cd0-4553-a4cf-98d64f120a45\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") "
Oct 30 20:32:15.762195 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.762049    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca\") pod \"599a2da6-8cd0-4553-a4cf-98d64f120a45\" (UID: \"599a2da6-8cd0-4553-a4cf-98d64f120a45\") "
Oct 30 20:32:15.762398 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.762372    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca" pod "599a2da6-8cd0-4553-a4cf-98d64f120a45" (UID: "599a2da6-8cd0-4553-a4cf-98d64f120a45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:32:15.763893 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.763866    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn" pod "599a2da6-8cd0-4553-a4cf-98d64f120a45" (UID: "599a2da6-8cd0-4553-a4cf-98d64f120a45"). InnerVolumeSpecName "kube-api-access-9rnvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:15.763976 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.763918    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca" pod "599a2da6-8cd0-4553-a4cf-98d64f120a45" (UID: "599a2da6-8cd0-4553-a4cf-98d64f120a45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:15.863378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.863353    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rnvn\" (UniqueName: \"kubernetes.io/projected/599a2da6-8cd0-4553-a4cf-98d64f120a45-kube-api-access-9rnvn\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:15.863378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.863375    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:15.863502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:15.863386    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a2da6-8cd0-4553-a4cf-98d64f120a45-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:16.461872 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:16.461842    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"
Oct 30 20:32:16.462053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:16.461839    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc" event={"ID":"599a2da6-8cd0-4553-a4cf-98d64f120a45","Type":"ContainerDied","Data":"0bc17d1e02599d7c237ef262fd072cb42a3f5496cb8a8a06a8d5b696dc39889e"}
Oct 30 20:32:16.462053 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:16.461967    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bc17d1e02599d7c237ef262fd072cb42a3f5496cb8a8a06a8d5b696dc39889e"
Oct 30 20:32:19.655402 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:32:19.655365    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:32:29.364222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.364188    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"]
Oct 30 20:32:29.364623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.364561    6058 cpu_manager.go:414] "containerMap: removing container" podUID="599a2da6-8cd0-4553-a4cf-98d64f120a45" containerName="revoke-keys"
Oct 30 20:32:29.364623 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.364574    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599a2da6-8cd0-4553-a4cf-98d64f120a45" containerName="revoke-keys"
Oct 30 20:32:29.364695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.364651    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="599a2da6-8cd0-4553-a4cf-98d64f120a45" containerName="revoke-keys"
Oct 30 20:32:29.367948 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.367929    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.376599 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.376575    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3f7c44\""
Oct 30 20:32:29.376774 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.376754    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bb158fe6-dockercfg-92cdx\""
Oct 30 20:32:29.378081 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.378057    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"]
Oct 30 20:32:29.388582 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.388555    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"]
Oct 30 20:32:29.392468 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.392449    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.402848 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.402816    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"]
Oct 30 20:32:29.472915 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.472873    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7kn\" (UniqueName: \"kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn\") pod \"payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.473141 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.473027    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z574\" (UniqueName: \"kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574\") pod \"payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.473220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.473195    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.473301 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.473249    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.573912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.573879    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.574117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.573928    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7kn\" (UniqueName: \"kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn\") pod \"payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.574117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.573958    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z574\" (UniqueName: \"kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574\") pod \"payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.574117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.574088    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.574634 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.574607    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.574836 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.574814    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.586798 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.586772    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7kn\" (UniqueName: \"kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn\") pod \"payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.586909 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.586804    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z574\" (UniqueName: \"kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574\") pod \"payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.681954 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.681891    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:29.707894 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.707862    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:29.966829 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:29.966800    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"]
Oct 30 20:32:29.968869 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:32:29.968839    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f1ea3d_28b8_4a08_9173_475146862145.slice/crio-4c31b4aa68d65bf7302864b82c3d9cf4091ade35d4b36c4e276f60d53773f398 WatchSource:0}: Error finding container 4c31b4aa68d65bf7302864b82c3d9cf4091ade35d4b36c4e276f60d53773f398: Status 404 returned error can't find the container with id 4c31b4aa68d65bf7302864b82c3d9cf4091ade35d4b36c4e276f60d53773f398
Oct 30 20:32:30.001071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.001046    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"]
Oct 30 20:32:30.005262 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:32:30.005230    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60354fba_3d0f_4d8f_80ef_67061dce6198.slice/crio-421788ac3d59d60757a15cebdc15ba26effdf168b599385953b80d26ed448dcd WatchSource:0}: Error finding container 421788ac3d59d60757a15cebdc15ba26effdf168b599385953b80d26ed448dcd: Status 404 returned error can't find the container with id 421788ac3d59d60757a15cebdc15ba26effdf168b599385953b80d26ed448dcd
Oct 30 20:32:30.529197 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.529163    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" event={"ID":"98f1ea3d-28b8-4a08-9173-475146862145","Type":"ContainerStarted","Data":"ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65"}
Oct 30 20:32:30.529789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.529227    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" event={"ID":"98f1ea3d-28b8-4a08-9173-475146862145","Type":"ContainerStarted","Data":"4c31b4aa68d65bf7302864b82c3d9cf4091ade35d4b36c4e276f60d53773f398"}
Oct 30 20:32:30.529789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.529264    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:30.530816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.530784    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" event={"ID":"60354fba-3d0f-4d8f-80ef-67061dce6198","Type":"ContainerStarted","Data":"d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908"}
Oct 30 20:32:30.530929 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.530831    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" event={"ID":"60354fba-3d0f-4d8f-80ef-67061dce6198","Type":"ContainerStarted","Data":"421788ac3d59d60757a15cebdc15ba26effdf168b599385953b80d26ed448dcd"}
Oct 30 20:32:30.530992 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.530929    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:30.549809 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.549766    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" podStartSLOduration=1.549747531 podStartE2EDuration="1.549747531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:30.548533503 +0000 UTC m=+1683.451953530" watchObservedRunningTime="2026-10-30 20:32:30.549747531 +0000 UTC m=+1683.453167566"
Oct 30 20:32:30.574851 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:30.574805    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" podStartSLOduration=1.5747910809999999 podStartE2EDuration="1.574791081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:30.573714223 +0000 UTC m=+1683.477134250" watchObservedRunningTime="2026-10-30 20:32:30.574791081 +0000 UTC m=+1683.478211108"
Oct 30 20:32:31.649158 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:31.649125    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:32:41.539049 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:41.538989    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:32:41.539533 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:41.539101    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:32:46.565488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:32:46.565447    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:32:49.656220 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:32:49.656181    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:33:02.462899 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.462858    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:33:02.815534 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.815454    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"]
Oct 30 20:33:02.819518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.819497    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:02.828681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.828211    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"]
Oct 30 20:33:02.830203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.830179    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:33:02.830439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.830283    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:02.830439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.830317    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:02.830439 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:02.830355    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:03.002559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.002520    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gp49\" (UniqueName: \"kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.002559 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.002561    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.002767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.002738    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.103868 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.103790    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp49\" (UniqueName: \"kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.103868 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.103831    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.104124 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.103949    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.104616 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.104590    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.106190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.106167    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.116563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.116539    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gp49\" (UniqueName: \"kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49\") pod \"maas-api-revoke-sub-5096c697-mnk8f\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.136311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.136285    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:03.422385 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.422356    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"]
Oct 30 20:33:03.424569 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:33:03.424540    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e614e4_a4d5_438c_8642_eee1ed46609b.slice/crio-c1e110bfceb2e47bccdeab16caee78968ea638307166491e92f054eeec330581 WatchSource:0}: Error finding container c1e110bfceb2e47bccdeab16caee78968ea638307166491e92f054eeec330581: Status 404 returned error can't find the container with id c1e110bfceb2e47bccdeab16caee78968ea638307166491e92f054eeec330581
Oct 30 20:33:03.705857 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:03.705776    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f" event={"ID":"d7e614e4-a4d5-438c-8642-eee1ed46609b","Type":"ContainerStarted","Data":"c1e110bfceb2e47bccdeab16caee78968ea638307166491e92f054eeec330581"}
Oct 30 20:33:04.711576 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:04.711536    6058 generic.go:356] "Generic (PLEG): container finished" podID="d7e614e4-a4d5-438c-8642-eee1ed46609b" containerID="760348bf2834d512928556ddb7f264c7fdb48efc8e256bb4f7395311154947d8" exitCode=0
Oct 30 20:33:04.711965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:04.711629    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f" event={"ID":"d7e614e4-a4d5-438c-8642-eee1ed46609b","Type":"ContainerDied","Data":"760348bf2834d512928556ddb7f264c7fdb48efc8e256bb4f7395311154947d8"}
Oct 30 20:33:05.840051 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:05.840027    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:06.032423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.032352    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca\") pod \"d7e614e4-a4d5-438c-8642-eee1ed46609b\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") "
Oct 30 20:33:06.032562 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.032471    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49\" (UniqueName: \"kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49\") pod \"d7e614e4-a4d5-438c-8642-eee1ed46609b\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") "
Oct 30 20:33:06.032613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.032556    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca\") pod \"d7e614e4-a4d5-438c-8642-eee1ed46609b\" (UID: \"d7e614e4-a4d5-438c-8642-eee1ed46609b\") "
Oct 30 20:33:06.032869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.032843    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca" pod "d7e614e4-a4d5-438c-8642-eee1ed46609b" (UID: "d7e614e4-a4d5-438c-8642-eee1ed46609b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:06.034299 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.034277    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca" pod "d7e614e4-a4d5-438c-8642-eee1ed46609b" (UID: "d7e614e4-a4d5-438c-8642-eee1ed46609b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:06.034421 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.034401    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49" pod "d7e614e4-a4d5-438c-8642-eee1ed46609b" (UID: "d7e614e4-a4d5-438c-8642-eee1ed46609b"). InnerVolumeSpecName "kube-api-access-2gp49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:06.133878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.133855    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:06.133878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.133878    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e614e4-a4d5-438c-8642-eee1ed46609b-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:06.134025 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.133888    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gp49\" (UniqueName: \"kubernetes.io/projected/d7e614e4-a4d5-438c-8642-eee1ed46609b-kube-api-access-2gp49\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:06.721314 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.721283    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"
Oct 30 20:33:06.721493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.721310    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f" event={"ID":"d7e614e4-a4d5-438c-8642-eee1ed46609b","Type":"ContainerDied","Data":"c1e110bfceb2e47bccdeab16caee78968ea638307166491e92f054eeec330581"}
Oct 30 20:33:06.721493 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:06.721367    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1e110bfceb2e47bccdeab16caee78968ea638307166491e92f054eeec330581"
Oct 30 20:33:13.761144 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.761052    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:33:13.850174 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.850138    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"]
Oct 30 20:33:13.850541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.850528    6058 cpu_manager.go:414] "containerMap: removing container" podUID="d7e614e4-a4d5-438c-8642-eee1ed46609b" containerName="revoke-keys"
Oct 30 20:33:13.850595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.850542    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e614e4-a4d5-438c-8642-eee1ed46609b" containerName="revoke-keys"
Oct 30 20:33:13.850633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.850619    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e614e4-a4d5-438c-8642-eee1ed46609b" containerName="revoke-keys"
Oct 30 20:33:13.853726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.853707    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:13.860119 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.860100    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:13.861655 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.861632    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:13.862167 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.862144    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:13.862265 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.862184    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:33:13.881806 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.881774    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"]
Oct 30 20:33:13.899326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.899291    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:13.899326 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.899325    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxb2d\" (UniqueName: \"kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:13.899561 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:13.899509    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.000366 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.000335    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.000565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.000391    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.000565 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.000410    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxb2d\" (UniqueName: \"kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.001172 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.001147    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.002522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.002503    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.015527 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.015471    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxb2d\" (UniqueName: \"kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.168397 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.168368    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:14.443894 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.443867    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"]
Oct 30 20:33:14.446243 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:33:14.446214    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc4b623_391e_4d40_924f_4ba2c528281e.slice/crio-572cb348616216d0b5d4fdf64d0625ccfb246c601f18780e7defae5dbff4af6d WatchSource:0}: Error finding container 572cb348616216d0b5d4fdf64d0625ccfb246c601f18780e7defae5dbff4af6d: Status 404 returned error can't find the container with id 572cb348616216d0b5d4fdf64d0625ccfb246c601f18780e7defae5dbff4af6d
Oct 30 20:33:14.761700 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:14.761623    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps" event={"ID":"ebc4b623-391e-4d40-924f-4ba2c528281e","Type":"ContainerStarted","Data":"572cb348616216d0b5d4fdf64d0625ccfb246c601f18780e7defae5dbff4af6d"}
Oct 30 20:33:15.768790 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:15.768708    6058 generic.go:356] "Generic (PLEG): container finished" podID="ebc4b623-391e-4d40-924f-4ba2c528281e" containerID="65fa3c64cbf40789e49ec79973f210c008dba885e3c6c229cad10c02bb7fc561" exitCode=0
Oct 30 20:33:15.768790 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:15.768764    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps" event={"ID":"ebc4b623-391e-4d40-924f-4ba2c528281e","Type":"ContainerDied","Data":"65fa3c64cbf40789e49ec79973f210c008dba885e3c6c229cad10c02bb7fc561"}
Oct 30 20:33:16.921423 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:16.921394    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:17.029972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.029884    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca\") pod \"ebc4b623-391e-4d40-924f-4ba2c528281e\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") "
Oct 30 20:33:17.030160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.030000    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca\") pod \"ebc4b623-391e-4d40-924f-4ba2c528281e\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") "
Oct 30 20:33:17.030160 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.030110    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d\" (UniqueName: \"kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d\") pod \"ebc4b623-391e-4d40-924f-4ba2c528281e\" (UID: \"ebc4b623-391e-4d40-924f-4ba2c528281e\") "
Oct 30 20:33:17.030418 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.030386    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca" pod "ebc4b623-391e-4d40-924f-4ba2c528281e" (UID: "ebc4b623-391e-4d40-924f-4ba2c528281e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:17.031977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.031954    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca" pod "ebc4b623-391e-4d40-924f-4ba2c528281e" (UID: "ebc4b623-391e-4d40-924f-4ba2c528281e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:17.032089 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.032052    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d" pod "ebc4b623-391e-4d40-924f-4ba2c528281e" (UID: "ebc4b623-391e-4d40-924f-4ba2c528281e"). InnerVolumeSpecName "kube-api-access-jxb2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:17.131102 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.131073    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:17.131102 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.131099    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxb2d\" (UniqueName: \"kubernetes.io/projected/ebc4b623-391e-4d40-924f-4ba2c528281e-kube-api-access-jxb2d\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:17.131271 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.131109    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc4b623-391e-4d40-924f-4ba2c528281e-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:17.779189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.779150    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps" event={"ID":"ebc4b623-391e-4d40-924f-4ba2c528281e","Type":"ContainerDied","Data":"572cb348616216d0b5d4fdf64d0625ccfb246c601f18780e7defae5dbff4af6d"}
Oct 30 20:33:17.779362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.779204    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="572cb348616216d0b5d4fdf64d0625ccfb246c601f18780e7defae5dbff4af6d"
Oct 30 20:33:17.779362 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:17.779169    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"
Oct 30 20:33:19.655267 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:33:19.655233    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:33:23.975594 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:23.975559    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"]
Oct 30 20:33:23.982672 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:23.982635    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"]
Oct 30 20:33:24.416243 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:24.416202    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"]
Oct 30 20:33:24.420037 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:24.420000    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f7c44-sqlps"]
Oct 30 20:33:25.211544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:25.211511    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc4b623-391e-4d40-924f-4ba2c528281e" path="/var/lib/kubelet/pods/ebc4b623-391e-4d40-924f-4ba2c528281e/volumes"
Oct 30 20:33:25.446945 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:25.446907    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:33:28.976832 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:28.976770    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerName="payload-processing" containerID="cri-o://d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908" gracePeriod=25
Oct 30 20:33:28.983275 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:28.983227    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" podUID="98f1ea3d-28b8-4a08-9173-475146862145" containerName="payload-pre-processing" containerID="cri-o://ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65" gracePeriod=25
Oct 30 20:33:31.537311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:31.537268    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:33:31.537686 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:31.537268    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" podUID="98f1ea3d-28b8-4a08-9173-475146862145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:33:38.945604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:38.945567    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:33:41.537659 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:41.537599    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:33:41.539372 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:41.539333    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" podUID="98f1ea3d-28b8-4a08-9173-475146862145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:33:42.202134 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:42.202094    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:42.202428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:42.202395    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:42.217131 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:42.217103    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:42.217329 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:42.217306    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:49.242750 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.242726    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:33:49.245688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.245669    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:33:49.287641 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.287603    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume\") pod \"60354fba-3d0f-4d8f-80ef-67061dce6198\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") "
Oct 30 20:33:49.287808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.287673    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574\" (UniqueName: \"kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574\") pod \"98f1ea3d-28b8-4a08-9173-475146862145\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") "
Oct 30 20:33:49.287808 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.287784    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume\") pod \"98f1ea3d-28b8-4a08-9173-475146862145\" (UID: \"98f1ea3d-28b8-4a08-9173-475146862145\") "
Oct 30 20:33:49.287936 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.287850    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn\" (UniqueName: \"kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn\") pod \"60354fba-3d0f-4d8f-80ef-67061dce6198\" (UID: \"60354fba-3d0f-4d8f-80ef-67061dce6198\") "
Oct 30 20:33:49.288129 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.288095    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume" pod "60354fba-3d0f-4d8f-80ef-67061dce6198" (UID: "60354fba-3d0f-4d8f-80ef-67061dce6198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:49.288322 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.288299    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60354fba-3d0f-4d8f-80ef-67061dce6198-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.288864 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.288837    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume" pod "98f1ea3d-28b8-4a08-9173-475146862145" (UID: "98f1ea3d-28b8-4a08-9173-475146862145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:49.290597 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.290576    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574" pod "98f1ea3d-28b8-4a08-9173-475146862145" (UID: "98f1ea3d-28b8-4a08-9173-475146862145"). InnerVolumeSpecName "kube-api-access-8z574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:49.290704 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.290681    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn" pod "60354fba-3d0f-4d8f-80ef-67061dce6198" (UID: "60354fba-3d0f-4d8f-80ef-67061dce6198"). InnerVolumeSpecName "kube-api-access-7z7kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:49.389083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.389042    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z574\" (UniqueName: \"kubernetes.io/projected/98f1ea3d-28b8-4a08-9173-475146862145-kube-api-access-8z574\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.389083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.389076    6058 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f1ea3d-28b8-4a08-9173-475146862145-plugins-config-volume\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.389083 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.389085    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7kn\" (UniqueName: \"kubernetes.io/projected/60354fba-3d0f-4d8f-80ef-67061dce6198-kube-api-access-7z7kn\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.658282 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:33:49.658244    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:33:49.972125 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.972038    6058 generic.go:356] "Generic (PLEG): container finished" podID="98f1ea3d-28b8-4a08-9173-475146862145" containerID="ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65" exitCode=0
Oct 30 20:33:49.972297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.972135    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" event={"ID":"98f1ea3d-28b8-4a08-9173-475146862145","Type":"ContainerDied","Data":"ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65"}
Oct 30 20:33:49.972297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.972157    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"
Oct 30 20:33:49.972297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.972216    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj" event={"ID":"98f1ea3d-28b8-4a08-9173-475146862145","Type":"ContainerDied","Data":"4c31b4aa68d65bf7302864b82c3d9cf4091ade35d4b36c4e276f60d53773f398"}
Oct 30 20:33:49.972297 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.972249    6058 scope.go:122] "RemoveContainer" containerID="ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65"
Oct 30 20:33:49.973581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.973558    6058 generic.go:356] "Generic (PLEG): container finished" podID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerID="d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908" exitCode=0
Oct 30 20:33:49.973684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.973619    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"
Oct 30 20:33:49.973684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.973630    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" event={"ID":"60354fba-3d0f-4d8f-80ef-67061dce6198","Type":"ContainerDied","Data":"d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908"}
Oct 30 20:33:49.973684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.973662    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk" event={"ID":"60354fba-3d0f-4d8f-80ef-67061dce6198","Type":"ContainerDied","Data":"421788ac3d59d60757a15cebdc15ba26effdf168b599385953b80d26ed448dcd"}
Oct 30 20:33:49.982896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.982877    6058 scope.go:122] "RemoveContainer" containerID="ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65"
Oct 30 20:33:49.983165 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:33:49.983145    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65\": container with ID starting with ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65 not found: ID does not exist" containerID="ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65"
Oct 30 20:33:49.983268 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.983171    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65"} err="failed to get container status \"ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65\": rpc error: code = NotFound desc = could not find container \"ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65\": container with ID starting with ee71cb0ddc8fb8208b07342815a654fa01510be1d6e4b56dcce792de527e6f65 not found: ID does not exist"
Oct 30 20:33:49.983268 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.983190    6058 scope.go:122] "RemoveContainer" containerID="d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908"
Oct 30 20:33:49.996252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:49.996233    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"]
Oct 30 20:33:50.000200 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:50.000180    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f7c44-7df646nrlj"]
Oct 30 20:33:50.000403 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:50.000390    6058 scope.go:122] "RemoveContainer" containerID="d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908"
Oct 30 20:33:50.000669 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:33:50.000654    6058 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908\": container with ID starting with d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908 not found: ID does not exist" containerID="d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908"
Oct 30 20:33:50.000714 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:50.000683    6058 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908"} err="failed to get container status \"d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908\": rpc error: code = NotFound desc = could not find container \"d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908\": container with ID starting with d573174409321eec921891353aba9e7d4c763410bff92b385a0a1de190cd6908 not found: ID does not exist"
Oct 30 20:33:50.012977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:50.012957    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"]
Oct 30 20:33:50.018086 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:50.018066    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f7c44-6c5b567c6cqtrk"]
Oct 30 20:33:51.211602 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:51.211568    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" path="/var/lib/kubelet/pods/60354fba-3d0f-4d8f-80ef-67061dce6198/volumes"
Oct 30 20:33:51.211995 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:51.211980    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f1ea3d-28b8-4a08-9173-475146862145" path="/var/lib/kubelet/pods/98f1ea3d-28b8-4a08-9173-475146862145/volumes"
Oct 30 20:33:52.039704 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.039669    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"]
Oct 30 20:33:52.040122 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040105    6058 cpu_manager.go:414] "containerMap: removing container" podUID="ebc4b623-391e-4d40-924f-4ba2c528281e" containerName="revoke-keys"
Oct 30 20:33:52.040122 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040121    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc4b623-391e-4d40-924f-4ba2c528281e" containerName="revoke-keys"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040132    6058 cpu_manager.go:414] "containerMap: removing container" podUID="98f1ea3d-28b8-4a08-9173-475146862145" containerName="payload-pre-processing"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040140    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f1ea3d-28b8-4a08-9173-475146862145" containerName="payload-pre-processing"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040156    6058 cpu_manager.go:414] "containerMap: removing container" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerName="payload-processing"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040161    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerName="payload-processing"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040228    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc4b623-391e-4d40-924f-4ba2c528281e" containerName="revoke-keys"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040236    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f1ea3d-28b8-4a08-9173-475146862145" containerName="payload-pre-processing"
Oct 30 20:33:52.040260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.040244    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="60354fba-3d0f-4d8f-80ef-67061dce6198" containerName="payload-processing"
Oct 30 20:33:52.044979 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.044963    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.052433 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.052408    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-99bbafd5-kserve-self-signed-certs\""
Oct 30 20:33:52.054230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.054207    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"]
Oct 30 20:33:52.112240 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.112212    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.112371 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.112273    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.112371 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.112355    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfgk\" (UniqueName: \"kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.112446 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.112399    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.112446 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.112418    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.112541 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.112519    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.213804 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.213768    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.213815    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfgk\" (UniqueName: \"kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.213881    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.213913    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.213970    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.214038    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214307 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.214238    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214639 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.214469    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.214687 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.214651    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.216482 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.216460    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.216738 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.216720    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.226850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.226827    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfgk\" (UniqueName: \"kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk\") pod \"e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") " pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.357956 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.357930    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:52.634189 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.634024    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"]
Oct 30 20:33:52.636977 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:33:52.636950    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a27b9e_bf35_4c1f_94de_4cdecc41484f.slice/crio-d7ff218161cae30a999b0e56f9925ed8978da28663bd71f0d56e970bbc92dcf7 WatchSource:0}: Error finding container d7ff218161cae30a999b0e56f9925ed8978da28663bd71f0d56e970bbc92dcf7: Status 404 returned error can't find the container with id d7ff218161cae30a999b0e56f9925ed8978da28663bd71f0d56e970bbc92dcf7
Oct 30 20:33:52.992120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.992025    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" event={"ID":"e5a27b9e-bf35-4c1f-94de-4cdecc41484f","Type":"ContainerStarted","Data":"4c31d4c116b1add133e3831d7496fade9016eabfe32bd3179b913270728d869e"}
Oct 30 20:33:52.992120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:52.992077    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" event={"ID":"e5a27b9e-bf35-4c1f-94de-4cdecc41484f","Type":"ContainerStarted","Data":"d7ff218161cae30a999b0e56f9925ed8978da28663bd71f0d56e970bbc92dcf7"}
Oct 30 20:33:53.359050 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:53.359019    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:33:53.634949 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:53.634825    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" podStartSLOduration=1.634805348 podStartE2EDuration="1.634805348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:53.012936537 +0000 UTC m=+1765.916356565" watchObservedRunningTime="2026-10-30 20:33:53.634805348 +0000 UTC m=+1766.538225375"
Oct 30 20:33:53.637001 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:53.636973    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:33:57.204030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:57.203104    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-ws6r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:57.204030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:57.203204    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-ws6r4" podUID="b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:57.224028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:57.220080    6058 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jrd57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:57.224028 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:33:57.220134    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jrd57" podUID="cad85ff2-3b19-4098-aeb6-50feed17e4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:34:02.359153 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:02.359121    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:34:02.371373 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:02.371345    6058 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:34:03.077155 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:03.077126    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:34:09.769782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.769739    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"]
Oct 30 20:34:09.773829 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.773804    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.778258 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.778233    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:09.780620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.780596    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:34:09.782971 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.782946    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"]
Oct 30 20:34:09.841588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.841558    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:09.841860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.841654    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24v4\" (UniqueName: \"kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.841860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.841711    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.841860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.841727    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:09.842089 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.841952    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.942502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.942465    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24v4\" (UniqueName: \"kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.942764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.942513    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.942764 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.942591    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.943222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.943194    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.944878 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.944858    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:09.955796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:09.955765    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24v4\" (UniqueName: \"kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4\") pod \"maas-api-revoke-sub-452b6d7b-fg9g8\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:10.091663 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:10.091593    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:10.161611 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:10.160805    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:34:10.368714 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:10.368690    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"]
Oct 30 20:34:10.370965 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:34:10.370929    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2cb6c8_0105_4dec_9b75_8b0c07916098.slice/crio-775d95511abe9eb0811c50cb9dc79f53d916fb024249f165b893e17a0cb86be5 WatchSource:0}: Error finding container 775d95511abe9eb0811c50cb9dc79f53d916fb024249f165b893e17a0cb86be5: Status 404 returned error can't find the container with id 775d95511abe9eb0811c50cb9dc79f53d916fb024249f165b893e17a0cb86be5
Oct 30 20:34:11.108820 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:11.108787    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8" event={"ID":"7e2cb6c8-0105-4dec-9b75-8b0c07916098","Type":"ContainerStarted","Data":"bc0af14d9441f9b7aac91ec38c76367c2377e4fe788bbdd7289bd2e1ae4fe3cd"}
Oct 30 20:34:11.109344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:11.108843    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8" event={"ID":"7e2cb6c8-0105-4dec-9b75-8b0c07916098","Type":"ContainerStarted","Data":"775d95511abe9eb0811c50cb9dc79f53d916fb024249f165b893e17a0cb86be5"}
Oct 30 20:34:11.129809 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:11.129741    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8" podStartSLOduration=2.129722728 podStartE2EDuration="2.129722728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:11.128545019 +0000 UTC m=+1784.031965045" watchObservedRunningTime="2026-10-30 20:34:11.129722728 +0000 UTC m=+1784.033142755"
Oct 30 20:34:12.135030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:12.131844    6058 generic.go:356] "Generic (PLEG): container finished" podID="7e2cb6c8-0105-4dec-9b75-8b0c07916098" containerID="bc0af14d9441f9b7aac91ec38c76367c2377e4fe788bbdd7289bd2e1ae4fe3cd" exitCode=0
Oct 30 20:34:12.135030 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:12.131932    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8" event={"ID":"7e2cb6c8-0105-4dec-9b75-8b0c07916098","Type":"ContainerDied","Data":"bc0af14d9441f9b7aac91ec38c76367c2377e4fe788bbdd7289bd2e1ae4fe3cd"}
Oct 30 20:34:13.567796 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.567508    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:13.743875 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.743765    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca\") pod \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") "
Oct 30 20:34:13.743875 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.743826    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca\") pod \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") "
Oct 30 20:34:13.744173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.743889    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4\" (UniqueName: \"kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4\") pod \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\" (UID: \"7e2cb6c8-0105-4dec-9b75-8b0c07916098\") "
Oct 30 20:34:13.744513 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.744480    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca" pod "7e2cb6c8-0105-4dec-9b75-8b0c07916098" (UID: "7e2cb6c8-0105-4dec-9b75-8b0c07916098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:13.746589 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.746564    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4" pod "7e2cb6c8-0105-4dec-9b75-8b0c07916098" (UID: "7e2cb6c8-0105-4dec-9b75-8b0c07916098"). InnerVolumeSpecName "kube-api-access-p24v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:13.746695 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.746646    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca" pod "7e2cb6c8-0105-4dec-9b75-8b0c07916098" (UID: "7e2cb6c8-0105-4dec-9b75-8b0c07916098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:13.844812 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.844773    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:13.845099 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.845065    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2cb6c8-0105-4dec-9b75-8b0c07916098-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:13.845221 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:13.845207    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24v4\" (UniqueName: \"kubernetes.io/projected/7e2cb6c8-0105-4dec-9b75-8b0c07916098-kube-api-access-p24v4\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:14.144535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:14.144497    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8" event={"ID":"7e2cb6c8-0105-4dec-9b75-8b0c07916098","Type":"ContainerDied","Data":"775d95511abe9eb0811c50cb9dc79f53d916fb024249f165b893e17a0cb86be5"}
Oct 30 20:34:14.144701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:14.144553    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="775d95511abe9eb0811c50cb9dc79f53d916fb024249f165b893e17a0cb86be5"
Oct 30 20:34:14.144701 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:14.144506    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg9g8"
Oct 30 20:34:19.655146 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:34:19.655103    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:34:21.941427 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:21.941390    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:34:22.803210 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:22.803158    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"]
Oct 30 20:34:25.922926 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.922887    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"]
Oct 30 20:34:25.923339 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.923313    6058 cpu_manager.go:414] "containerMap: removing container" podUID="7e2cb6c8-0105-4dec-9b75-8b0c07916098" containerName="revoke-keys"
Oct 30 20:34:25.923339 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.923325    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2cb6c8-0105-4dec-9b75-8b0c07916098" containerName="revoke-keys"
Oct 30 20:34:25.923437 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.923389    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2cb6c8-0105-4dec-9b75-8b0c07916098" containerName="revoke-keys"
Oct 30 20:34:25.926585 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.926558    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:25.930985 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.930958    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:25.931139 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.931087    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:34:25.932070 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.932050    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:25.935208 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.935186    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:25.942168 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.942139    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"]
Oct 30 20:34:25.998086 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.998047    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbc5m\" (UniqueName: \"kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:25.998284 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.998211    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:25.998364 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:25.998285    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.099508 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.099469    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.099681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.099522    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.099681 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.099575    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbc5m\" (UniqueName: \"kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.100222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.100200    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.101797 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.101756    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.112278 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.112249    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbc5m\" (UniqueName: \"kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m\") pod \"maas-api-revoke-sub-3e31a506-fcfc4\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.242668 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.242591    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:26.551354 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:26.551201    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"]
Oct 30 20:34:26.554708 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:34:26.554678    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4ca7e4_00ec_43d2_b9b6_d3f25b776600.slice/crio-25a19877db72810411b0cafa71102b9074554b23a1e2f81f80b17670a7a0a133 WatchSource:0}: Error finding container 25a19877db72810411b0cafa71102b9074554b23a1e2f81f80b17670a7a0a133: Status 404 returned error can't find the container with id 25a19877db72810411b0cafa71102b9074554b23a1e2f81f80b17670a7a0a133
Oct 30 20:34:27.251624 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:27.251582    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4" event={"ID":"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600","Type":"ContainerStarted","Data":"3bb8d36666312b8af414a4b69fc69660e10efe28d3de7662e0fba77e52281434"}
Oct 30 20:34:27.253132 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:27.253104    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4" event={"ID":"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600","Type":"ContainerStarted","Data":"25a19877db72810411b0cafa71102b9074554b23a1e2f81f80b17670a7a0a133"}
Oct 30 20:34:27.301044 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:27.299676    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4" podStartSLOduration=2.299657578 podStartE2EDuration="2.299657578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:27.298814065 +0000 UTC m=+1800.202234091" watchObservedRunningTime="2026-10-30 20:34:27.299657578 +0000 UTC m=+1800.203077607"
Oct 30 20:34:28.278377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:28.278341    6058 generic.go:356] "Generic (PLEG): container finished" podID="dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" containerID="3bb8d36666312b8af414a4b69fc69660e10efe28d3de7662e0fba77e52281434" exitCode=0
Oct 30 20:34:28.278985 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:28.278959    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4" event={"ID":"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600","Type":"ContainerDied","Data":"3bb8d36666312b8af414a4b69fc69660e10efe28d3de7662e0fba77e52281434"}
Oct 30 20:34:28.530138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:28.522965    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:34:28.734048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:28.730515    6058 scope.go:122] "RemoveContainer" containerID="55532976c2c69f36c6370cdfcc8f900897b86e4516651b17cb36e36539927a50"
Oct 30 20:34:29.413409 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.413378    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:29.465036 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.464987    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca\") pod \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") "
Oct 30 20:34:29.465179 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.465075    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca\") pod \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") "
Oct 30 20:34:29.465260 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.465182    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m\" (UniqueName: \"kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m\") pod \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\" (UID: \"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600\") "
Oct 30 20:34:29.465515 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.465484    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca" pod "dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" (UID: "dd4ca7e4-00ec-43d2-b9b6-d3f25b776600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:29.467156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.467135    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m" pod "dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" (UID: "dd4ca7e4-00ec-43d2-b9b6-d3f25b776600"). InnerVolumeSpecName "kube-api-access-tbc5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:29.467383 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.467368    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca" pod "dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" (UID: "dd4ca7e4-00ec-43d2-b9b6-d3f25b776600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:29.566538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.566460    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbc5m\" (UniqueName: \"kubernetes.io/projected/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-kube-api-access-tbc5m\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:29.566538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.566491    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:29.566538 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:29.566503    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ca7e4-00ec-43d2-b9b6-d3f25b776600-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:30.289462 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:30.289429    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4"
Oct 30 20:34:30.289613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:30.289455    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fcfc4" event={"ID":"dd4ca7e4-00ec-43d2-b9b6-d3f25b776600","Type":"ContainerDied","Data":"25a19877db72810411b0cafa71102b9074554b23a1e2f81f80b17670a7a0a133"}
Oct 30 20:34:30.289613 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:30.289514    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a19877db72810411b0cafa71102b9074554b23a1e2f81f80b17670a7a0a133"
Oct 30 20:34:36.751048 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:36.751001    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:34:37.840143 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:37.840049    6058 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" podUID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" containerName="main" containerID="cri-o://4c31d4c116b1add133e3831d7496fade9016eabfe32bd3179b913270728d869e" gracePeriod=45
Oct 30 20:34:38.070532 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.070124    6058 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" podUID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.134:8000/ready\": dial tcp 10.133.0.134:8000: connect: connection refused"
Oct 30 20:34:38.350021 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.349969    6058 generic.go:356] "Generic (PLEG): container finished" podID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" containerID="4c31d4c116b1add133e3831d7496fade9016eabfe32bd3179b913270728d869e" exitCode=0
Oct 30 20:34:38.350156 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.350037    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" event={"ID":"e5a27b9e-bf35-4c1f-94de-4cdecc41484f","Type":"ContainerDied","Data":"4c31d4c116b1add133e3831d7496fade9016eabfe32bd3179b913270728d869e"}
Oct 30 20:34:38.435109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.435027    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:34:38.474311 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.474278    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir\") pod \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") "
Oct 30 20:34:38.474572 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.474552    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache\") pod \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") "
Oct 30 20:34:38.474752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.474737    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm\") pod \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") "
Oct 30 20:34:38.474932 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.474917    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home\") pod \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") "
Oct 30 20:34:38.488109 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.488082    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk\" (UniqueName: \"kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk\") pod \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") "
Oct 30 20:34:38.488287 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.488270    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs\") pod \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\" (UID: \"e5a27b9e-bf35-4c1f-94de-4cdecc41484f\") "
Oct 30 20:34:38.489620 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.478421    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home" pod "e5a27b9e-bf35-4c1f-94de-4cdecc41484f" (UID: "e5a27b9e-bf35-4c1f-94de-4cdecc41484f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:38.489783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.478650    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir" pod "e5a27b9e-bf35-4c1f-94de-4cdecc41484f" (UID: "e5a27b9e-bf35-4c1f-94de-4cdecc41484f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:38.501220 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.493093    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache" pod "e5a27b9e-bf35-4c1f-94de-4cdecc41484f" (UID: "e5a27b9e-bf35-4c1f-94de-4cdecc41484f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:38.509344 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.509299    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk" pod "e5a27b9e-bf35-4c1f-94de-4cdecc41484f" (UID: "e5a27b9e-bf35-4c1f-94de-4cdecc41484f"). InnerVolumeSpecName "kube-api-access-5mfgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:38.535219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.520161    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs" pod "e5a27b9e-bf35-4c1f-94de-4cdecc41484f" (UID: "e5a27b9e-bf35-4c1f-94de-4cdecc41484f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:38.535219 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.529107    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm" pod "e5a27b9e-bf35-4c1f-94de-4cdecc41484f" (UID: "e5a27b9e-bf35-4c1f-94de-4cdecc41484f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:38.590027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.589608    6058 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-model-cache\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:38.590027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.589647    6058 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-dshm\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:38.590027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.589662    6058 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-home\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:38.590027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.589676    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfgk\" (UniqueName: \"kubernetes.io/projected/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-kube-api-access-5mfgk\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:38.590027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.589692    6058 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tls-certs\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:38.590027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:38.589706    6058 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a27b9e-bf35-4c1f-94de-4cdecc41484f-tmp-dir\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:39.357901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:39.357864    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz" event={"ID":"e5a27b9e-bf35-4c1f-94de-4cdecc41484f","Type":"ContainerDied","Data":"d7ff218161cae30a999b0e56f9925ed8978da28663bd71f0d56e970bbc92dcf7"}
Oct 30 20:34:39.357901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:39.357882    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"
Oct 30 20:34:39.358540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:39.357922    6058 scope.go:122] "RemoveContainer" containerID="4c31d4c116b1add133e3831d7496fade9016eabfe32bd3179b913270728d869e"
Oct 30 20:34:39.378555 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:39.378513    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"]
Oct 30 20:34:39.380441 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:39.380416    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-99bbafd5-kserve-84b664d9b9-47cqz"]
Oct 30 20:34:41.226031 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:41.225301    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" path="/var/lib/kubelet/pods/e5a27b9e-bf35-4c1f-94de-4cdecc41484f/volumes"
Oct 30 20:34:41.612027 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:41.610754    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:34:46.025520 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:46.025486    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"]
Oct 30 20:34:46.028834 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:46.028812    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f44cd44-fwmlm"]
Oct 30 20:34:47.211177 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.211147    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d988fcc-ac23-4fd8-bc93-c132ed17b327" path="/var/lib/kubelet/pods/2d988fcc-ac23-4fd8-bc93-c132ed17b327/volumes"
Oct 30 20:34:47.503422 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503345    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"]
Oct 30 20:34:47.503815 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503795    6058 cpu_manager.go:414] "containerMap: removing container" podUID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" containerName="main"
Oct 30 20:34:47.503916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503817    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" containerName="main"
Oct 30 20:34:47.503916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503836    6058 cpu_manager.go:414] "containerMap: removing container" podUID="dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" containerName="revoke-keys"
Oct 30 20:34:47.503916 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503844    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" containerName="revoke-keys"
Oct 30 20:34:47.504104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503971    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a27b9e-bf35-4c1f-94de-4cdecc41484f" containerName="main"
Oct 30 20:34:47.504104 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.503987    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4ca7e4-00ec-43d2-b9b6-d3f25b776600" containerName="revoke-keys"
Oct 30 20:34:47.508486 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.508464    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.516078 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.515999    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:34:47.516173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.516155    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:47.516235 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.516153    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"]
Oct 30 20:34:47.516235 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.516218    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:47.516360 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.516281    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:47.601726 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.601689    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.601892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.601742    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbdk\" (UniqueName: \"kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.601892 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.601872    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.702654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.702619    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.702826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.702668    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbdk\" (UniqueName: \"kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.702826 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.702715    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.703258 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.703229    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.704944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.704922    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.716055 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.716030    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbdk\" (UniqueName: \"kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk\") pod \"maas-api-revoke-sub-15260e3b-fvrzl\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:47.822649 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:47.822580    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:48.098101 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:48.097946    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"]
Oct 30 20:34:48.101539 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:34:48.101509    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda769b6e4_d291_46d3_93f1_f0801d635ef6.slice/crio-57d49ebcd800d99c043af6a7fec56e49bb8d00a7456a2936aa85dbc2af41b213 WatchSource:0}: Error finding container 57d49ebcd800d99c043af6a7fec56e49bb8d00a7456a2936aa85dbc2af41b213: Status 404 returned error can't find the container with id 57d49ebcd800d99c043af6a7fec56e49bb8d00a7456a2936aa85dbc2af41b213
Oct 30 20:34:48.430669 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:48.430635    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl" event={"ID":"a769b6e4-d291-46d3-93f1-f0801d635ef6","Type":"ContainerStarted","Data":"3eb737aa8ecbb2e58f8ad4bbd30f3f0133ca44b7e0470ffd991b86411c5bc079"}
Oct 30 20:34:48.431073 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:48.430683    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl" event={"ID":"a769b6e4-d291-46d3-93f1-f0801d635ef6","Type":"ContainerStarted","Data":"57d49ebcd800d99c043af6a7fec56e49bb8d00a7456a2936aa85dbc2af41b213"}
Oct 30 20:34:48.452203 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:48.452157    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl" podStartSLOduration=1.452145028 podStartE2EDuration="1.452145028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:48.449753381 +0000 UTC m=+1821.353173410" watchObservedRunningTime="2026-10-30 20:34:48.452145028 +0000 UTC m=+1821.355565054"
Oct 30 20:34:49.437604 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:49.437567    6058 generic.go:356] "Generic (PLEG): container finished" podID="a769b6e4-d291-46d3-93f1-f0801d635ef6" containerID="3eb737aa8ecbb2e58f8ad4bbd30f3f0133ca44b7e0470ffd991b86411c5bc079" exitCode=0
Oct 30 20:34:49.437985 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:49.437660    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl" event={"ID":"a769b6e4-d291-46d3-93f1-f0801d635ef6","Type":"ContainerDied","Data":"3eb737aa8ecbb2e58f8ad4bbd30f3f0133ca44b7e0470ffd991b86411c5bc079"}
Oct 30 20:34:49.655628 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:34:49.655589    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:34:50.574233 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.574208    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:50.731567 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.731502    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk\" (UniqueName: \"kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk\") pod \"a769b6e4-d291-46d3-93f1-f0801d635ef6\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") "
Oct 30 20:34:50.731714 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.731570    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca\") pod \"a769b6e4-d291-46d3-93f1-f0801d635ef6\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") "
Oct 30 20:34:50.731714 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.731656    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca\") pod \"a769b6e4-d291-46d3-93f1-f0801d635ef6\" (UID: \"a769b6e4-d291-46d3-93f1-f0801d635ef6\") "
Oct 30 20:34:50.732038 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.731994    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca" pod "a769b6e4-d291-46d3-93f1-f0801d635ef6" (UID: "a769b6e4-d291-46d3-93f1-f0801d635ef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:50.733581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.733558    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca" pod "a769b6e4-d291-46d3-93f1-f0801d635ef6" (UID: "a769b6e4-d291-46d3-93f1-f0801d635ef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:50.733727 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.733626    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk" pod "a769b6e4-d291-46d3-93f1-f0801d635ef6" (UID: "a769b6e4-d291-46d3-93f1-f0801d635ef6"). InnerVolumeSpecName "kube-api-access-xqbdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:50.832392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.832361    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:50.832392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.832391    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769b6e4-d291-46d3-93f1-f0801d635ef6-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:50.832558 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:50.832408    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqbdk\" (UniqueName: \"kubernetes.io/projected/a769b6e4-d291-46d3-93f1-f0801d635ef6-kube-api-access-xqbdk\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:51.447428 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:51.447390    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl" event={"ID":"a769b6e4-d291-46d3-93f1-f0801d635ef6","Type":"ContainerDied","Data":"57d49ebcd800d99c043af6a7fec56e49bb8d00a7456a2936aa85dbc2af41b213"}
Oct 30 20:34:51.447581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:51.447442    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57d49ebcd800d99c043af6a7fec56e49bb8d00a7456a2936aa85dbc2af41b213"
Oct 30 20:34:51.447581 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:51.447406    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15260e3b-fvrzl"
Oct 30 20:34:58.329901 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:34:58.329866    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:35:08.026488 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:08.026456    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:35:13.134252 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:13.134213    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:35:17.028782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.028742    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"]
Oct 30 20:35:17.036651 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.036618    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6hgl"]
Oct 30 20:35:17.213595 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.213558    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc01c4a2-9a5a-406e-847c-2bd770d09fec" path="/var/lib/kubelet/pods/cc01c4a2-9a5a-406e-847c-2bd770d09fec/volumes"
Oct 30 20:35:17.424965 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.424934    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"]
Oct 30 20:35:17.425411 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.425385    6058 cpu_manager.go:414] "containerMap: removing container" podUID="a769b6e4-d291-46d3-93f1-f0801d635ef6" containerName="revoke-keys"
Oct 30 20:35:17.425515 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.425413    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a769b6e4-d291-46d3-93f1-f0801d635ef6" containerName="revoke-keys"
Oct 30 20:35:17.425574 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.425545    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="a769b6e4-d291-46d3-93f1-f0801d635ef6" containerName="revoke-keys"
Oct 30 20:35:17.430034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.429992    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.438074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.438023    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:17.438074 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.438023    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:35:17.438241 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.438087    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:17.438443 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.438368    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:17.438782 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.438761    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"]
Oct 30 20:35:17.571688 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.571651    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.571855 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.571713    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.571899 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.571863    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9s5\" (UniqueName: \"kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.673789 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.673745    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9s5\" (UniqueName: \"kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.673986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.673828    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.673986 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.673880    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.674497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.674472    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.676331 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.676273    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.687191 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.687167    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9s5\" (UniqueName: \"kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5\") pod \"maas-api-revoke-sub-f03629db-44sqs\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:17.746542 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:17.746517    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:18.023497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:18.023468    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"]
Oct 30 20:35:18.025327 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:35:18.025291    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d90382c_cb2c_4bdf_a0b5_9ca16875ed18.slice/crio-479279235eb69f73a66242a6334616c1576cb02ba45867f8a66fc7146380ea65 WatchSource:0}: Error finding container 479279235eb69f73a66242a6334616c1576cb02ba45867f8a66fc7146380ea65: Status 404 returned error can't find the container with id 479279235eb69f73a66242a6334616c1576cb02ba45867f8a66fc7146380ea65
Oct 30 20:35:18.649935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:18.649854    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs" event={"ID":"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18","Type":"ContainerStarted","Data":"eafd24411b332de1bf092bfec6bb7d76b243fdab775e2f9318193f113f13f154"}
Oct 30 20:35:18.649935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:18.649910    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs" event={"ID":"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18","Type":"ContainerStarted","Data":"479279235eb69f73a66242a6334616c1576cb02ba45867f8a66fc7146380ea65"}
Oct 30 20:35:18.670222 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:18.670176    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs" podStartSLOduration=1.670162264 podStartE2EDuration="1.670162264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:18.669180599 +0000 UTC m=+1851.572600628" watchObservedRunningTime="2026-10-30 20:35:18.670162264 +0000 UTC m=+1851.573582290"
Oct 30 20:35:19.655271 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:35:19.655236    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:35:19.657377 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:19.657348    6058 generic.go:356] "Generic (PLEG): container finished" podID="9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" containerID="eafd24411b332de1bf092bfec6bb7d76b243fdab775e2f9318193f113f13f154" exitCode=0
Oct 30 20:35:19.657507 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:19.657430    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs" event={"ID":"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18","Type":"ContainerDied","Data":"eafd24411b332de1bf092bfec6bb7d76b243fdab775e2f9318193f113f13f154"}
Oct 30 20:35:20.787236 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.787213    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:20.900588 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.900559    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5\" (UniqueName: \"kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5\") pod \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") "
Oct 30 20:35:20.900761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.900677    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca\") pod \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") "
Oct 30 20:35:20.900761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.900717    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca\") pod \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\" (UID: \"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18\") "
Oct 30 20:35:20.901085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.901059    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca" pod "9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" (UID: "9d90382c-cb2c-4bdf-a0b5-9ca16875ed18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:20.902698 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.902675    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5" pod "9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" (UID: "9d90382c-cb2c-4bdf-a0b5-9ca16875ed18"). InnerVolumeSpecName "kube-api-access-zs9s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:20.902773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:20.902693    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca" pod "9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" (UID: "9d90382c-cb2c-4bdf-a0b5-9ca16875ed18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:21.001590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:21.001527    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs9s5\" (UniqueName: \"kubernetes.io/projected/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-kube-api-access-zs9s5\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:21.001590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:21.001559    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:21.001590 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:21.001570    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90382c-cb2c-4bdf-a0b5-9ca16875ed18-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:21.668563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:21.668532    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs"
Oct 30 20:35:21.668563 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:21.668543    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-44sqs" event={"ID":"9d90382c-cb2c-4bdf-a0b5-9ca16875ed18","Type":"ContainerDied","Data":"479279235eb69f73a66242a6334616c1576cb02ba45867f8a66fc7146380ea65"}
Oct 30 20:35:21.668756 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:21.668589    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479279235eb69f73a66242a6334616c1576cb02ba45867f8a66fc7146380ea65"
Oct 30 20:35:27.951415 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:27.951375    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:35:28.808137 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:28.808107    6058 scope.go:122] "RemoveContainer" containerID="8ab5ccebd7a234bcc115fdd3b830677a18ffbe3f1d5af87d9f1e62c222f985f8"
Oct 30 20:35:33.073425 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.073390    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"]
Oct 30 20:35:33.073811 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.073764    6058 cpu_manager.go:414] "containerMap: removing container" podUID="9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" containerName="revoke-keys"
Oct 30 20:35:33.073811 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.073774    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" containerName="revoke-keys"
Oct 30 20:35:33.073886 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.073843    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d90382c-cb2c-4bdf-a0b5-9ca16875ed18" containerName="revoke-keys"
Oct 30 20:35:33.078649 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.078626    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.088320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.088297    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:33.088477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.088327    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:33.088477 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.088370    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:33.088670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.088412    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:35:33.088912 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.088885    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"]
Oct 30 20:35:33.237192 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.237151    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.237376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.237207    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htll\" (UniqueName: \"kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.237376 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.237289    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.338120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.338053    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htll\" (UniqueName: \"kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.338120 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.338088    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.338320 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.338145    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.338748 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.338725    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.340392 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.340369    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.351230 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.351209    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htll\" (UniqueName: \"kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll\") pod \"maas-api-revoke-sub-0165d7d2-zq4cg\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.392717 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.392693    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:33.669734 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.669705    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"]
Oct 30 20:35:33.672757 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:35:33.672730    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859ea9aa_1944_4a81_9da0_a8cc86326a5e.slice/crio-43f72de8e66b5015e7ffb13fbd292d133a99d96c8ad876de707931b981f3cece WatchSource:0}: Error finding container 43f72de8e66b5015e7ffb13fbd292d133a99d96c8ad876de707931b981f3cece: Status 404 returned error can't find the container with id 43f72de8e66b5015e7ffb13fbd292d133a99d96c8ad876de707931b981f3cece
Oct 30 20:35:33.737646 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:33.737617    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg" event={"ID":"859ea9aa-1944-4a81-9da0-a8cc86326a5e","Type":"ContainerStarted","Data":"43f72de8e66b5015e7ffb13fbd292d133a99d96c8ad876de707931b981f3cece"}
Oct 30 20:35:34.743570 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:34.743496    6058 generic.go:356] "Generic (PLEG): container finished" podID="859ea9aa-1944-4a81-9da0-a8cc86326a5e" containerID="74e9ee439b2ff69109430eb3321770b089fb0f06517a125b39c08347528ae935" exitCode=0
Oct 30 20:35:34.743902 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:34.743559    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg" event={"ID":"859ea9aa-1944-4a81-9da0-a8cc86326a5e","Type":"ContainerDied","Data":"74e9ee439b2ff69109430eb3321770b089fb0f06517a125b39c08347528ae935"}
Oct 30 20:35:35.873836 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:35.873811    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:36.065316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.065243    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll\" (UniqueName: \"kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll\") pod \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") "
Oct 30 20:35:36.065316 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.065282    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca\") pod \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") "
Oct 30 20:35:36.065551 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.065359    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca\") pod \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\" (UID: \"859ea9aa-1944-4a81-9da0-a8cc86326a5e\") "
Oct 30 20:35:36.065761 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.065738    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca" pod "859ea9aa-1944-4a81-9da0-a8cc86326a5e" (UID: "859ea9aa-1944-4a81-9da0-a8cc86326a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:36.067317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.067288    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca" pod "859ea9aa-1944-4a81-9da0-a8cc86326a5e" (UID: "859ea9aa-1944-4a81-9da0-a8cc86326a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:36.067317 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.067300    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll" pod "859ea9aa-1944-4a81-9da0-a8cc86326a5e" (UID: "859ea9aa-1944-4a81-9da0-a8cc86326a5e"). InnerVolumeSpecName "kube-api-access-4htll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:36.166731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.166706    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:36.166731 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.166731    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htll\" (UniqueName: \"kubernetes.io/projected/859ea9aa-1944-4a81-9da0-a8cc86326a5e-kube-api-access-4htll\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:36.166860 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.166747    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859ea9aa-1944-4a81-9da0-a8cc86326a5e-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:36.754357 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.754328    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg"
Oct 30 20:35:36.754500 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.754371    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zq4cg" event={"ID":"859ea9aa-1944-4a81-9da0-a8cc86326a5e","Type":"ContainerDied","Data":"43f72de8e66b5015e7ffb13fbd292d133a99d96c8ad876de707931b981f3cece"}
Oct 30 20:35:36.754500 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:36.754416    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f72de8e66b5015e7ffb13fbd292d133a99d96c8ad876de707931b981f3cece"
Oct 30 20:35:44.019728 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.019689    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"]
Oct 30 20:35:44.020245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.020127    6058 cpu_manager.go:414] "containerMap: removing container" podUID="859ea9aa-1944-4a81-9da0-a8cc86326a5e" containerName="revoke-keys"
Oct 30 20:35:44.020245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.020144    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859ea9aa-1944-4a81-9da0-a8cc86326a5e" containerName="revoke-keys"
Oct 30 20:35:44.020245 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.020212    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="859ea9aa-1944-4a81-9da0-a8cc86326a5e" containerName="revoke-keys"
Oct 30 20:35:44.023596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.023571    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.028351 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.028321    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9jg\" (UniqueName: \"kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.028513 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.028375    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.028513 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.028478    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.029066 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.029044    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:44.029310 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.029288    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:44.029447 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.029361    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbqtx\""
Oct 30 20:35:44.029447 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.029371    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:44.038773 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.038748    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"]
Oct 30 20:35:44.129977 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.129937    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9jg\" (UniqueName: \"kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.130173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.130039    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.130173 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.130101    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.130781 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.130756    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.132456 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.132434    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.143499 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.143479    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9jg\" (UniqueName: \"kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg\") pod \"maas-api-revoke-sub-6b19338f-xbnrh\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.337957 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.337874    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:44.616494 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.616432    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"]
Oct 30 20:35:44.618687 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:35:44.618660    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302b32f7_5c80_4e87_a802_8f09ace01523.slice/crio-c7eefa073cebff8661eddc0210710996252ebd8f6a099cab26b1a118d9cc55da WatchSource:0}: Error finding container c7eefa073cebff8661eddc0210710996252ebd8f6a099cab26b1a118d9cc55da: Status 404 returned error can't find the container with id c7eefa073cebff8661eddc0210710996252ebd8f6a099cab26b1a118d9cc55da
Oct 30 20:35:44.793088 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:44.793046    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh" event={"ID":"302b32f7-5c80-4e87-a802-8f09ace01523","Type":"ContainerStarted","Data":"c7eefa073cebff8661eddc0210710996252ebd8f6a099cab26b1a118d9cc55da"}
Oct 30 20:35:45.805619 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:45.805543    6058 generic.go:356] "Generic (PLEG): container finished" podID="302b32f7-5c80-4e87-a802-8f09ace01523" containerID="e5f77e83f1f52555c75a455deea4ff9b8d11a56c4aae805ce04b91f41e16d0d7" exitCode=0
Oct 30 20:35:45.805972 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:45.805623    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh" event={"ID":"302b32f7-5c80-4e87-a802-8f09ace01523","Type":"ContainerDied","Data":"e5f77e83f1f52555c75a455deea4ff9b8d11a56c4aae805ce04b91f41e16d0d7"}
Oct 30 20:35:46.946983 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:46.946957    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:47.054937 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.054911    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg\" (UniqueName: \"kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg\") pod \"302b32f7-5c80-4e87-a802-8f09ace01523\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") "
Oct 30 20:35:47.055106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.054974    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca\") pod \"302b32f7-5c80-4e87-a802-8f09ace01523\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") "
Oct 30 20:35:47.055106 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.054994    6058 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca\") pod \"302b32f7-5c80-4e87-a802-8f09ace01523\" (UID: \"302b32f7-5c80-4e87-a802-8f09ace01523\") "
Oct 30 20:35:47.055378 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.055347    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca" pod "302b32f7-5c80-4e87-a802-8f09ace01523" (UID: "302b32f7-5c80-4e87-a802-8f09ace01523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:47.057087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.057060    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg" pod "302b32f7-5c80-4e87-a802-8f09ace01523" (UID: "302b32f7-5c80-4e87-a802-8f09ace01523"). InnerVolumeSpecName "kube-api-access-zw9jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:47.057087 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.057072    6058 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca" pod "302b32f7-5c80-4e87-a802-8f09ace01523" (UID: "302b32f7-5c80-4e87-a802-8f09ace01523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:47.155896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.155866    6058 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9jg\" (UniqueName: \"kubernetes.io/projected/302b32f7-5c80-4e87-a802-8f09ace01523-kube-api-access-zw9jg\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:47.155896 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.155891    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-openshift-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:47.156068 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.155902    6058 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b32f7-5c80-4e87-a802-8f09ace01523-maas-api-xks-ca\") on node \"ip-10-0-141-189.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:47.817858 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.817824    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh" event={"ID":"302b32f7-5c80-4e87-a802-8f09ace01523","Type":"ContainerDied","Data":"c7eefa073cebff8661eddc0210710996252ebd8f6a099cab26b1a118d9cc55da"}
Oct 30 20:35:47.818056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.817877    6058 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7eefa073cebff8661eddc0210710996252ebd8f6a099cab26b1a118d9cc55da"
Oct 30 20:35:47.818056 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:47.817833    6058 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xbnrh"
Oct 30 20:35:49.655022 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:35:49.654973    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:35:54.849544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:54.849510    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2vml"]
Oct 30 20:35:55.473032 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:35:55.472193    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f7d6c68cd-mbtq4_d9b85d4a-009b-42d2-96a8-08391753c3d1/authorino/0.log"
Oct 30 20:36:08.638841 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:08.638771    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp_9ef85828-b3d6-4b6f-b0be-a061b570503f/copy-manifests/0.log"
Oct 30 20:36:08.659503 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:08.659476    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp_9ef85828-b3d6-4b6f-b0be-a061b570503f/manager/0.log"
Oct 30 20:36:10.552850 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:10.552818    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zq4cg_859ea9aa-1944-4a81-9da0-a8cc86326a5e/revoke-keys/0.log"
Oct 30 20:36:10.749544 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:10.749514    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-15260e3b-fvrzl_a769b6e4-d291-46d3-93f1-f0801d635ef6/revoke-keys/0.log"
Oct 30 20:36:10.938984 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:10.938957    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-fcfc4_dd4ca7e4-00ec-43d2-b9b6-d3f25b776600/revoke-keys/0.log"
Oct 30 20:36:11.125282 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:11.125252    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-fg9g8_7e2cb6c8-0105-4dec-9b75-8b0c07916098/revoke-keys/0.log"
Oct 30 20:36:11.313672 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:11.313602    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5096c697-mnk8f_d7e614e4-a4d5-438c-8642-eee1ed46609b/revoke-keys/0.log"
Oct 30 20:36:11.505501 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:11.505476    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xbnrh_302b32f7-5c80-4e87-a802-8f09ace01523/revoke-keys/0.log"
Oct 30 20:36:11.690497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:11.690475    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-xskkc_599a2da6-8cd0-4553-a4cf-98d64f120a45/revoke-keys/0.log"
Oct 30 20:36:11.884851 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:11.884823    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-xdxwt_046f5cc6-ad54-4852-b1a3-b661bed25c6d/revoke-keys/0.log"
Oct 30 20:36:12.078631 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:12.078557    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-44sqs_9d90382c-cb2c-4bdf-a0b5-9ca16875ed18/revoke-keys/0.log"
Oct 30 20:36:13.888635 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:13.888603    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f7d6c68cd-mbtq4_d9b85d4a-009b-42d2-96a8-08391753c3d1/authorino/0.log"
Oct 30 20:36:14.322355 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:14.322280    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-mdr4m_bc8a5270-0a47-47db-9d1a-1ec0dad81ce5/manager/0.log"
Oct 30 20:36:14.716596 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:14.716566    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-j6tq5_10ba5c8d-a75a-42e0-a5f1-57bb3abf312e/manager/0.log"
Oct 30 20:36:14.976545 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:14.976459    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h2vml_a35d36de-bc9b-4740-b19e-9efb37df2d2f/limitador/0.log"
Oct 30 20:36:16.635654 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:16.635614    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d477969dd-rzcf7_d3ec1597-b02b-469a-b2b8-7c1a085b76f9/kube-auth-proxy/0.log"
Oct 30 20:36:17.447495 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:17.447463    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jrd57_cad85ff2-3b19-4098-aeb6-50feed17e4aa/istio-proxy/0.log"
Oct 30 20:36:17.638944 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:17.638905    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ls9cm_dc5e977f-8820-4bd8-a733-4c35785d3fc5/istio-proxy/0.log"
Oct 30 20:36:18.624556 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:18.624523    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ws6r4_b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a/istio-proxy/0.log"
Oct 30 20:36:19.655211 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:36:19.655178    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:36:20.394788 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:20.394760    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5qft9_75e73891-2ed1-4066-83f7-4f9a9d384f66/main/0.log"
Oct 30 20:36:20.783237 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:20.783165    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-m8gtk_6a19f9af-b5c9-4122-a0e4-20f215eef257/main/0.log"
Oct 30 20:36:21.174190 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:21.174146    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j_1a27023a-06b5-4452-bfa2-294228fd0143/main/0.log"
Oct 30 20:36:21.364182 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:21.364149    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n_acb420cc-2386-4cca-b3ae-a53a9fb370e6/main/0.log"
Oct 30 20:36:22.314838 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:22.314805    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f7d6c68cd-mbtq4_d9b85d4a-009b-42d2-96a8-08391753c3d1/authorino/0.log"
Oct 30 20:36:28.831417 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:28.831385    6058 scope.go:122] "RemoveContainer" containerID="807aca43892ff542424f6f8aa69bf2e79430875a4b01dd3ee45d35f812324d2e"
Oct 30 20:36:28.839873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:28.839840    6058 scope.go:122] "RemoveContainer" containerID="e6cf85daa313ae2722b3c54f58256b36268b8cb38b62796b57809477e085f59b"
Oct 30 20:36:28.847497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:28.847479    6058 scope.go:122] "RemoveContainer" containerID="2092d0d9e37c8ab8a3e0b68289460810db593a7cefccdc65c814f3bda796ea7d"
Oct 30 20:36:33.766401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:33.766373    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp_9ef85828-b3d6-4b6f-b0be-a061b570503f/copy-manifests/0.log"
Oct 30 20:36:33.788400 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:33.788375    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp_9ef85828-b3d6-4b6f-b0be-a061b570503f/manager/0.log"
Oct 30 20:36:35.382633 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:35.382604    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zq4cg_859ea9aa-1944-4a81-9da0-a8cc86326a5e/revoke-keys/0.log"
Oct 30 20:36:35.569955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:35.569900    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-15260e3b-fvrzl_a769b6e4-d291-46d3-93f1-f0801d635ef6/revoke-keys/0.log"
Oct 30 20:36:35.762479 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:35.762414    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-fcfc4_dd4ca7e4-00ec-43d2-b9b6-d3f25b776600/revoke-keys/0.log"
Oct 30 20:36:35.939294 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:35.939267    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-fg9g8_7e2cb6c8-0105-4dec-9b75-8b0c07916098/revoke-keys/0.log"
Oct 30 20:36:36.114928 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:36.114908    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5096c697-mnk8f_d7e614e4-a4d5-438c-8642-eee1ed46609b/revoke-keys/0.log"
Oct 30 20:36:36.302352 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:36.302324    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xbnrh_302b32f7-5c80-4e87-a802-8f09ace01523/revoke-keys/0.log"
Oct 30 20:36:36.484679 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:36.484606    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-xskkc_599a2da6-8cd0-4553-a4cf-98d64f120a45/revoke-keys/0.log"
Oct 30 20:36:36.658830 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:36.658804    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-xdxwt_046f5cc6-ad54-4852-b1a3-b661bed25c6d/revoke-keys/0.log"
Oct 30 20:36:36.844436 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:36.844362    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-44sqs_9d90382c-cb2c-4bdf-a0b5-9ca16875ed18/revoke-keys/0.log"
Oct 30 20:36:38.182722 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:38.182691    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f7d6c68cd-mbtq4_d9b85d4a-009b-42d2-96a8-08391753c3d1/authorino/0.log"
Oct 30 20:36:38.587987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:38.587906    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-mdr4m_bc8a5270-0a47-47db-9d1a-1ec0dad81ce5/manager/0.log"
Oct 30 20:36:38.936486 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:38.936459    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-j6tq5_10ba5c8d-a75a-42e0-a5f1-57bb3abf312e/manager/0.log"
Oct 30 20:36:39.161115 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:39.161089    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h2vml_a35d36de-bc9b-4740-b19e-9efb37df2d2f/limitador/0.log"
Oct 30 20:36:40.469145 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:40.469114    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d477969dd-rzcf7_d3ec1597-b02b-469a-b2b8-7c1a085b76f9/kube-auth-proxy/0.log"
Oct 30 20:36:41.226553 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:41.226527    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jrd57_cad85ff2-3b19-4098-aeb6-50feed17e4aa/istio-proxy/0.log"
Oct 30 20:36:41.408151 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:41.408114    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ls9cm_dc5e977f-8820-4bd8-a733-4c35785d3fc5/istio-proxy/0.log"
Oct 30 20:36:42.357935 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:42.357899    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ws6r4_b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a/istio-proxy/0.log"
Oct 30 20:36:43.642122 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:43.642094    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5qft9_75e73891-2ed1-4066-83f7-4f9a9d384f66/main/0.log"
Oct 30 20:36:44.010987 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:44.010912    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-m8gtk_6a19f9af-b5c9-4122-a0e4-20f215eef257/main/0.log"
Oct 30 20:36:44.372586 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:44.372561    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2cx4j_1a27023a-06b5-4452-bfa2-294228fd0143/main/0.log"
Oct 30 20:36:44.552886 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:36:44.552848    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fg97n_acb420cc-2386-4cca-b3ae-a53a9fb370e6/main/0.log"
Oct 30 20:36:49.655813 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:36:49.655774    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:37:00.030540 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:37:00.030504    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"]
Oct 30 20:37:00.033816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:37:00.033788    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xdxwt"]
Oct 30 20:37:01.210869 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:37:01.210822    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046f5cc6-ad54-4852-b1a3-b661bed25c6d" path="/var/lib/kubelet/pods/046f5cc6-ad54-4852-b1a3-b661bed25c6d/volumes"
Oct 30 20:37:16.024873 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:37:16.024838    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"]
Oct 30 20:37:16.028767 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:37:16.028739    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xskkc"]
Oct 30 20:37:17.212042 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:37:17.211990    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599a2da6-8cd0-4553-a4cf-98d64f120a45" path="/var/lib/kubelet/pods/599a2da6-8cd0-4553-a4cf-98d64f120a45/volumes"
Oct 30 20:37:19.657569 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:37:19.657532    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:37:49.654486 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:37:49.654402    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:38:06.024518 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:06.024485    6058 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"]
Oct 30 20:38:06.029670 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:06.029639    6058 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5096c697-mnk8f"]
Oct 30 20:38:07.217451 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:07.217417    6058 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e614e4-a4d5-438c-8642-eee1ed46609b" path="/var/lib/kubelet/pods/d7e614e4-a4d5-438c-8642-eee1ed46609b/volumes"
Oct 30 20:38:11.999138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:11.999100    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rnhfw/must-gather-279nr"]
Oct 30 20:38:11.999642 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:11.999468    6058 cpu_manager.go:414] "containerMap: removing container" podUID="302b32f7-5c80-4e87-a802-8f09ace01523" containerName="revoke-keys"
Oct 30 20:38:11.999642 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:11.999479    6058 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302b32f7-5c80-4e87-a802-8f09ace01523" containerName="revoke-keys"
Oct 30 20:38:11.999642 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:11.999561    6058 memory_manager.go:365] "RemoveStaleState removing state" podUID="302b32f7-5c80-4e87-a802-8f09ace01523" containerName="revoke-keys"
Oct 30 20:38:12.002918 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.002900    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.008431 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.008404    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnhfw\"/\"kube-root-ca.crt\""
Oct 30 20:38:12.009401 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.009379    6058 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rnhfw\"/\"default-dockercfg-5cxtm\""
Oct 30 20:38:12.009483 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.009383    6058 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnhfw\"/\"openshift-service-ca.crt\""
Oct 30 20:38:12.028879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.028852    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnhfw/must-gather-279nr"]
Oct 30 20:38:12.119799 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.119767    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2095887b-ca08-43e6-b2a8-7d4d53fcd300-must-gather-output\") pod \"must-gather-279nr\" (UID: \"2095887b-ca08-43e6-b2a8-7d4d53fcd300\") " pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.119955 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.119877    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvt2\" (UniqueName: \"kubernetes.io/projected/2095887b-ca08-43e6-b2a8-7d4d53fcd300-kube-api-access-xrvt2\") pod \"must-gather-279nr\" (UID: \"2095887b-ca08-43e6-b2a8-7d4d53fcd300\") " pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.220900 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.220867    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvt2\" (UniqueName: \"kubernetes.io/projected/2095887b-ca08-43e6-b2a8-7d4d53fcd300-kube-api-access-xrvt2\") pod \"must-gather-279nr\" (UID: \"2095887b-ca08-43e6-b2a8-7d4d53fcd300\") " pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.221085 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.220922    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2095887b-ca08-43e6-b2a8-7d4d53fcd300-must-gather-output\") pod \"must-gather-279nr\" (UID: \"2095887b-ca08-43e6-b2a8-7d4d53fcd300\") " pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.221224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.221207    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2095887b-ca08-43e6-b2a8-7d4d53fcd300-must-gather-output\") pod \"must-gather-279nr\" (UID: \"2095887b-ca08-43e6-b2a8-7d4d53fcd300\") " pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.250730 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.250676    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvt2\" (UniqueName: \"kubernetes.io/projected/2095887b-ca08-43e6-b2a8-7d4d53fcd300-kube-api-access-xrvt2\") pod \"must-gather-279nr\" (UID: \"2095887b-ca08-43e6-b2a8-7d4d53fcd300\") " pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.316552 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.316528    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhfw/must-gather-279nr"
Oct 30 20:38:12.590535 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.590503    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnhfw/must-gather-279nr"]
Oct 30 20:38:12.593233 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:38:12.593206    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2095887b_ca08_43e6_b2a8_7d4d53fcd300.slice/crio-31f8c8032cfc4fd94468629a59d5ffce9504c06dbadb05069caeaf5bc6e2b841 WatchSource:0}: Error finding container 31f8c8032cfc4fd94468629a59d5ffce9504c06dbadb05069caeaf5bc6e2b841: Status 404 returned error can't find the container with id 31f8c8032cfc4fd94468629a59d5ffce9504c06dbadb05069caeaf5bc6e2b841
Oct 30 20:38:12.595185 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:12.595163    6058 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:38:13.538062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:13.537946    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhfw/must-gather-279nr" event={"ID":"2095887b-ca08-43e6-b2a8-7d4d53fcd300","Type":"ContainerStarted","Data":"a886fe819fcf1e4c417e33ba15d512dbdecb923eac829bc21b55ebb3b8cc36fe"}
Oct 30 20:38:13.538062 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:13.538022    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhfw/must-gather-279nr" event={"ID":"2095887b-ca08-43e6-b2a8-7d4d53fcd300","Type":"ContainerStarted","Data":"31f8c8032cfc4fd94468629a59d5ffce9504c06dbadb05069caeaf5bc6e2b841"}
Oct 30 20:38:14.546164 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:14.546129    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhfw/must-gather-279nr" event={"ID":"2095887b-ca08-43e6-b2a8-7d4d53fcd300","Type":"ContainerStarted","Data":"a0baf5bacb28209778df88be7b75ae04610e5d5a5463c9658012835e6c1fe35e"}
Oct 30 20:38:14.568323 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:14.567463    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rnhfw/must-gather-279nr" podStartSLOduration=2.790608431 podStartE2EDuration="3.56744204s" totalImagesPullingTime="776.833609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:14.565232824 +0000 UTC m=+2027.468652852" watchObservedRunningTime="2026-10-30 20:38:14.56744204 +0000 UTC m=+2027.470862067"
Oct 30 20:38:15.372224 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:15.372175    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sxthj_2b57959a-5bcb-422a-bb6b-1d237cae31fb/global-pull-secret-syncer/0.log"
Oct 30 20:38:15.624064 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:15.623950    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ltkv6_a98a09f0-d4a5-41b5-8af3-06d949d06104/konnectivity-agent/0.log"
Oct 30 20:38:15.728649 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:15.728616    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-189.us-east-2.compute.internal_d8aa2772f25e2a2c0dd3af5f9c8bf9b9/haproxy/0.log"
Oct 30 20:38:19.803917 ip-10-0-141-189 kubenswrapper[6058]: E1030 20:38:19.803866    6058 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxgkl" containerName="extractor"
Oct 30 20:38:23.166166 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:23.166132    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f7d6c68cd-mbtq4_d9b85d4a-009b-42d2-96a8-08391753c3d1/authorino/0.log"
Oct 30 20:38:23.247497 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:23.247269    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-mdr4m_bc8a5270-0a47-47db-9d1a-1ec0dad81ce5/manager/0.log"
Oct 30 20:38:23.381865 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:23.381829    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-j6tq5_10ba5c8d-a75a-42e0-a5f1-57bb3abf312e/manager/0.log"
Oct 30 20:38:23.406539 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:23.406503    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h2vml_a35d36de-bc9b-4740-b19e-9efb37df2d2f/limitador/0.log"
Oct 30 20:38:24.937879 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:24.937847    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64d548bdbc-6fsb4_bf9894b4-5224-45ae-b205-86ad336937bb/metrics-server/0.log"
Oct 30 20:38:25.283920 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.283840    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vccbk_a572a882-20a9-49fb-92cb-1f1d618ed9b0/node-exporter/0.log"
Oct 30 20:38:25.318575 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.318544    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vccbk_a572a882-20a9-49fb-92cb-1f1d618ed9b0/kube-rbac-proxy/0.log"
Oct 30 20:38:25.347610 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.347581    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vccbk_a572a882-20a9-49fb-92cb-1f1d618ed9b0/init-textfile/0.log"
Oct 30 20:38:25.377388 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.377353    6058 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"]
Oct 30 20:38:25.384454 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.384426    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.391560 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.391527    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"]
Oct 30 20:38:25.466510 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.466472    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-podres\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.466697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.466532    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-proc\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.466697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.466568    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-lib-modules\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.466697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.466624    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5nn\" (UniqueName: \"kubernetes.io/projected/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-kube-api-access-6b5nn\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.466697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.466660    6058 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-sys\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.501126 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.501053    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/prometheus/0.log"
Oct 30 20:38:25.535234 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.535156    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/config-reloader/0.log"
Oct 30 20:38:25.567432 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567399    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-podres\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567440    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-proc\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567462    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-lib-modules\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567506    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5nn\" (UniqueName: \"kubernetes.io/projected/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-kube-api-access-6b5nn\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567542    6058 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-sys\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567544    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-podres\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567566    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-lib-modules\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567547    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-proc\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.567617 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.567612    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-sys\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.569822 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.569795    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/thanos-sidecar/0.log"
Oct 30 20:38:25.580820 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.580793    6058 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5nn\" (UniqueName: \"kubernetes.io/projected/1a67ef40-9996-45a3-9bad-bdd92d9dedbb-kube-api-access-6b5nn\") pod \"perf-node-gather-daemonset-bk56n\" (UID: \"1a67ef40-9996-45a3-9bad-bdd92d9dedbb\") " pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.597138 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.597104    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/kube-rbac-proxy-web/0.log"
Oct 30 20:38:25.624522 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.624488    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/kube-rbac-proxy/0.log"
Oct 30 20:38:25.659886 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.659854    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/kube-rbac-proxy-thanos/0.log"
Oct 30 20:38:25.688783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.688751    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4451f2a7-0e2b-4764-a083-8ae831378bb2/init-config-reloader/0.log"
Oct 30 20:38:25.701063 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.701034    6058 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:25.727520 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.727487    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-9v99w_d14bfe40-c6ab-48ff-a292-e67fc11dd857/prometheus-operator/0.log"
Oct 30 20:38:25.755573 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.755540    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-9v99w_d14bfe40-c6ab-48ff-a292-e67fc11dd857/kube-rbac-proxy/0.log"
Oct 30 20:38:25.827783 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.827668    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7564f9d8cc-bmrwv_fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/telemeter-client/0.log"
Oct 30 20:38:25.869697 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.869647    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7564f9d8cc-bmrwv_fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/reload/0.log"
Oct 30 20:38:25.896605 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.896576    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7564f9d8cc-bmrwv_fcb3ce4a-1765-4b82-8f0f-fd0107e77a7a/kube-rbac-proxy/0.log"
Oct 30 20:38:25.933169 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.933138    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7c4bc5db-qk2dh_d9f5607f-deae-4832-b1ef-1ee7f095ca16/thanos-query/0.log"
Oct 30 20:38:25.965709 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.965623    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7c4bc5db-qk2dh_d9f5607f-deae-4832-b1ef-1ee7f095ca16/kube-rbac-proxy-web/0.log"
Oct 30 20:38:25.986752 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.986720    6058 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"]
Oct 30 20:38:25.990948 ip-10-0-141-189 kubenswrapper[6058]: W1030 20:38:25.990921    6058 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a67ef40_9996_45a3_9bad_bdd92d9dedbb.slice/crio-a29388144bda1c0dd52fa3aa143d618de6ad5740b658952253691ba2fb92329a WatchSource:0}: Error finding container a29388144bda1c0dd52fa3aa143d618de6ad5740b658952253691ba2fb92329a: Status 404 returned error can't find the container with id a29388144bda1c0dd52fa3aa143d618de6ad5740b658952253691ba2fb92329a
Oct 30 20:38:25.994022 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:25.993984    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7c4bc5db-qk2dh_d9f5607f-deae-4832-b1ef-1ee7f095ca16/kube-rbac-proxy/0.log"
Oct 30 20:38:26.028204 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.028179    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7c4bc5db-qk2dh_d9f5607f-deae-4832-b1ef-1ee7f095ca16/prom-label-proxy/0.log"
Oct 30 20:38:26.060739 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.060712    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7c4bc5db-qk2dh_d9f5607f-deae-4832-b1ef-1ee7f095ca16/kube-rbac-proxy-rules/0.log"
Oct 30 20:38:26.089889 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.089810    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7c4bc5db-qk2dh_d9f5607f-deae-4832-b1ef-1ee7f095ca16/kube-rbac-proxy-metrics/0.log"
Oct 30 20:38:26.681025 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.680714    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n" event={"ID":"1a67ef40-9996-45a3-9bad-bdd92d9dedbb","Type":"ContainerStarted","Data":"811b481b5a48e64de5500fd73373ca8c765db0367efef4b803b2733d11544f78"}
Oct 30 20:38:26.681025 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.680774    6058 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n" event={"ID":"1a67ef40-9996-45a3-9bad-bdd92d9dedbb","Type":"ContainerStarted","Data":"a29388144bda1c0dd52fa3aa143d618de6ad5740b658952253691ba2fb92329a"}
Oct 30 20:38:26.681025 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.680910    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:26.704116 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:26.704059    6058 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n" podStartSLOduration=1.704040087 podStartE2EDuration="1.704040087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:26.702294157 +0000 UTC m=+2039.605714185" watchObservedRunningTime="2026-10-30 20:38:26.704040087 +0000 UTC m=+2039.607460114"
Oct 30 20:38:28.883925 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:28.883893    6058 scope.go:122] "RemoveContainer" containerID="4d93737b8b0589169e4d51821137371bd29af8f7baeb31570e97c989413a5551"
Oct 30 20:38:28.897098 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:28.897070    6058 scope.go:122] "RemoveContainer" containerID="ca2cdb5db63f828d56dfec22dc5cd7da12d5c3e9f076183c82a4883525629623"
Oct 30 20:38:30.312712 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:30.312687    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-68nkf_c4caf5a9-050f-4f30-a76d-6f0bff4b4409/dns/0.log"
Oct 30 20:38:30.347690 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:30.347663    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-68nkf_c4caf5a9-050f-4f30-a76d-6f0bff4b4409/kube-rbac-proxy/0.log"
Oct 30 20:38:30.570386 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:30.570307    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tdk9m_e282920c-4244-4ffd-87a8-a21d8c96d575/dns-node-resolver/0.log"
Oct 30 20:38:31.245823 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:31.245792    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nbkgn_fc397c8b-8550-4340-a2db-91263d9fcec0/node-ca/0.log"
Oct 30 20:38:32.698933 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:32.698901    6058 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rnhfw/perf-node-gather-daemonset-bk56n"
Oct 30 20:38:33.664272 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:33.664236    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d477969dd-rzcf7_d3ec1597-b02b-469a-b2b8-7c1a085b76f9/kube-auth-proxy/0.log"
Oct 30 20:38:33.813365 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:33.813323    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jrd57_cad85ff2-3b19-4098-aeb6-50feed17e4aa/istio-proxy/0.log"
Oct 30 20:38:33.874117 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:33.874061    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ls9cm_dc5e977f-8820-4bd8-a733-4c35785d3fc5/istio-proxy/0.log"
Oct 30 20:38:34.060071 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:34.057121    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ws6r4_b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a/istio-proxy/0.log"
Oct 30 20:38:35.860502 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:35.860471    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z6nqg_e9cf5ad2-9b7d-40da-a3ea-e10d4bb6a5c9/serve-healthcheck-canary/0.log"
Oct 30 20:38:36.682657 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:36.682627    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxgkl_df9edbdb-60a5-479c-a0de-c63c1725725e/kube-rbac-proxy/0.log"
Oct 30 20:38:36.712059 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:36.712032    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxgkl_df9edbdb-60a5-479c-a0de-c63c1725725e/exporter/0.log"
Oct 30 20:38:36.747684 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:36.747622    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxgkl_df9edbdb-60a5-479c-a0de-c63c1725725e/extractor/0.log"
Oct 30 20:38:37.192369 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:37.192338    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d477969dd-rzcf7_d3ec1597-b02b-469a-b2b8-7c1a085b76f9/kube-auth-proxy/0.log"
Oct 30 20:38:37.281159 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:37.281127    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jrd57_cad85ff2-3b19-4098-aeb6-50feed17e4aa/istio-proxy/0.log"
Oct 30 20:38:37.304305 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:37.304250    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ls9cm_dc5e977f-8820-4bd8-a733-4c35785d3fc5/istio-proxy/0.log"
Oct 30 20:38:37.398803 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:37.398770    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ws6r4_b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a/istio-proxy/0.log"
Oct 30 20:38:40.132034 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:40.131978    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp_9ef85828-b3d6-4b6f-b0be-a061b570503f/manager/0.log"
Oct 30 20:38:40.159768 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:40.159742    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfd5cbdbb-fktdp_9ef85828-b3d6-4b6f-b0be-a061b570503f/copy-manifests/0.log"
Oct 30 20:38:40.613343 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:40.613313    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d477969dd-rzcf7_d3ec1597-b02b-469a-b2b8-7c1a085b76f9/kube-auth-proxy/0.log"
Oct 30 20:38:40.707816 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:40.707784    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jrd57_cad85ff2-3b19-4098-aeb6-50feed17e4aa/istio-proxy/0.log"
Oct 30 20:38:40.739804 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:40.739771    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ls9cm_dc5e977f-8820-4bd8-a733-4c35785d3fc5/istio-proxy/0.log"
Oct 30 20:38:40.854231 ip-10-0-141-189 kubenswrapper[6058]: I1030 20:38:40.854187    6058 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-ws6r4_b51e8d7e-f49e-4c44-9f8e-ce56a09c8a2a/istio-proxy/0.log"